12--16 TB Storage Server - Questions on RAID, hardware, OS

[buckcm]

Check the last post first!

I also posted this thread here: http://www.tomshardware.com/forum/273194-31-storage-server-questions-raid-hardware#t2006250.

This is my first time posting on Tom's. I built my own gaming computer in July of 2008 for my birthday as a high schooler. I've done some smaller projects along the way, and I have been looking at computer parts for 2 years now (started to become thrilled just looking at video cards at TigerDirect) and Tom's since 1 1/2 years ago. I've been reading most of the front page articles for a couple months now and check Tom's everyday.


A friend of mine shoots a lot of video at 1080p and racks up 1 GB for every 1 minute of footage. In just a short hour, he has 60 GB of data. He wants something like 16 GB of storage. I tried to find the storage system he was going to buy; that's not important, but I know it was 16 TB of storage for $2500. I know I can do that for cheaper, so I want to build him a server.

My best stab at this is to buy a motherboard with 10 SATA connections. In it I want to put 10 1.5 TB hard drives for a (theoretical) total of 15 TB which is very close to his want of 16 TB.

Motherboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16813131326

Processor: http://www.newegg.com/Product/Product.aspx?Item=N82E16819115056

RAM: http://www.newegg.com/Product/Product.aspx?Item=N82E16820148163

Hard Drives: I am going to use (8) 2 TB hard drives now.

Case: Antec 1200

Video Card: http://www.newegg.com/Product/Product.aspx?Item=N82E16814127436&cm_re=4350-_-14-127-436-_-Product

Power Supply: http://www.newegg.com/Product/Product.aspx?Item=N82E16817341019&cm_re=ocz_700w-_-17-341-019-_-Product

Disk Drive: I'll just a DVD drive and will connect it via IDE (not very important).


Reasons for the parts:

I don't think this computer needs a crazy processor though the above dual core should be sufficient; furthermore, a C2D (or better) is necessary for any motherboards with 8+ SATA connections.

2 GB of RAM is OK, right? What about 4 GB for an extra $30?

I think that 700w is OK for all the hard drives (I did check Newegg's wattage calculator which is probably a bit pessimistic, though for this system it was under 600w). They should be WD Green or the Samsung 5400 RPM.

The Antec 1200 will house 10 hard drives nicely, don't you think?

The motherboard, again, has 10 SATA connections, gigabit ethernet, and RAID support. My friend needs to access the video files when he edits them on his MacBook Pro. This also has gigabit ethernet which is important for connecting to the sever quickly. Backup is another important thing for all this video. I believe RAID 5 is best in this situation; it will take 15 TB and make 10 TB so that any one disk can fail. This motherboard should be able to take all 10 hard drives and make them into one disk with a RAID 5, right?

Now, that storage center I mentioned that had 16 TB of storage cost $2500. I've seen units which cost just that much with no hard drives included. Even if that's the case, I will be well under $2500 with this 15 TB setup (16 with an external, so add $100 or so).

Cost Analysis:

120 (processor)
180 (motherboard)
~1000 (10 1.5 TB hard drives - notice that deal for the Samsung 1.5 TB hard drive was $95, a deal I found on Slickdeals.net)
40 (2 GB of RAM - add $30 for 4 GB)
75 (700w OCZ power supply before rebate)
35 (4350 before rebate)
140 Antec 1200 (I can save $30 if I get the 1200 with the same Samsung I found a deal on)
$20 for extra SATA cables
100 (external 1 TB HD)

I am getting about $1710. Add $30 for shipping maybe. I don't plan on ordering the hard drives until I find Green 1.5 TB for $100 or less (I already saw one today on Slickdeals so this shouldn't be a problem).

Anyways, that is substantially less than $2500 of the aforementioned storage center... I don't even know if that includes hard drives. Thoughts?

Forgot to ask: Can/should I use Linux for this (to save money)? Should I connect the server with Gigabit ethernet to my friend's MacBook Pro through a Linksys router or the sort? Are 1.5 TB hard drives good for RAID? I found that the Seagates may not be the best but is this true of all 1.5 TB hard drives?


And I just read the sticky...

APPROXIMATE PURCHASE DATE: Uh, as soon I get the go-ahead. Late October, early November 2009

SYSTEM USAGE FROM MOST TO LEAST IMPORTANT: Server

PARTS NOT REQUIRED: (e.g.: keyboard, mouse, monitor, speakers, OS): I'm pretty sure it just needs to hook up to a MacBook.

PREFERRED WEBSITE: I live in the US. Newegg is good though I do check around.

PARTS PREFERENCES:above

OVERCLOCKING: Not important.

MONITOR RESOLUTION: I have a Samsung 23". My friend has 2 24" monitors I believe at 1920x1080. I think his will be like mine. I have two computers hooked up to my monitor, one via VGA and the other DVI (thus the 4350 I chose with both).

ADDITIONAL COMMENTS: above

 

[sminlal]

Here are a few quick comments:

1) If you go to the Asus web site and look at the P5Q premium board, you'll see that 6 of the SATA ports are on one storage controller, and 4 are on another. So you couldn't put 10 drives into a single RAID set using the Intel RAID drivers (which is what most people do). You could do it at the Windows level, but I don't think you'd be able to boot from it if you did that.

2) If you're looking to put 10 x 1.5TB drives into RAID and get 15TB of storage out of it, then you must be thinking of using RAID 0. This is a bad idea for that many drives. With 10 drives, you have ten times the likelihood of a drive failure as with a single drive, and if any drive fails, you'll loose all of the data on the entire RAID set. You probably want at least RAID 5, although again it's not a great idea with that many drives.

3) Unless you can find a motherboard with GPT support, you won't be able to boot off a 15TB drive (2TB is the limit for MBR partitions). For that reason, and just for general manageability, it would probably be an good idea to put the OS on a separate volume - that would mean more than 10 drives.

4) If you're ever going to edit that HD video then you'll probably want more than a dual-core machine.

5) It would probably be a good idea to think about an external cabinet or perhaps a NAS for that many drives. You might be able to get them all into an Antec 1200 case (can't really comment on that), but it seems to me that you might have some challenges to keep both them and the CPU and graphics card all cool inside the same case without having so many fans screaming so loudly that you can't hear the video.

6) With that much data, you're going to have some real challenges making backups. You'd better include that in your plans.

 

[sub mesa]

Why not go for a NAS instead? With using FreeNAS you can save yourself the cost of a RAID controller and have reliable network-based storage. You do need gigabit ofcourse, and using another protocol than CIFS is recommended.

FreeNAS is very easy to configure, even if you've never touched anything else than Windows in your life. After booting the ISO and installing, you can do all configuring via web-interface from your windows pc. A more serious setup would be either Linux/BSD or OpenSolaris to run using ZFS filesystem. But before getting into this, decide whether you want a NAS (seperate system for storage) or direct-attached storage.

 

[buckcm]

OK. I got more details about the whole ordeal. This server needs to connect to a MacPro. Will FreeNAS work with Macs?

What options does that leave me with? Someone just tell me what to do as far as FreeNAS goes. I read about it, and it sounds good. Should I still do some sort of RAID (hardware)? Also, check out the ASUS website here: http://usa.asus.com/product.aspx?P_ID=QPhR6dGjcvkYnazE. The motherboard's SATA ports are clearly in 6 and 4. What options are at hand for a hardware RAID via the motherboard's controllers (I don't want to buy anything extra to add cost).

I'm also going to use (8) 2 TB hard drives now. The motherboard I chose, the Asus P5Q Premium, has 10 SATA ports but they are separated in 4 and 6 (http://usa.asus.com/product.aspx?P_ID=QPhR6dGjcvkYnazE).

Now I'm wondering how to connect the server to this MacPro. Should I use Firewire?

Here is what I'm competing with:http://www.lacie.com/us/products/product.htm?pid=11261.



I understand how RAIDs work. I don't want a 0 for sure. I don't know if (2) 0s would work with one mirroring the other?

I forgot to add: I am going to have (8) 2 TB hard drives as well as a smaller boot drive now.

 

[sminlal]

Sorry, can't help you with your Mac questions...

You can use what's called "RAID 0+1" or sometimes called "RAID 10" to both stripe and mirror drives. But mirroring means that you only get half of the space available - if you buy 8 x 2TB drives then you'll only have 8TB of space available.

RAID 5 gives you more usable capacity (14GB usable with 8 x 2TB drives), but has very poor write performance and increasing risk of data loss with sets of several or more disks, particularly for very large capacity disks (see: this Wikipedia section)

 

[HappyBison]

FreeNAS is a network OS, it will work with any OS which has the ability to talk to the network. In short, the answer is yes, it will work with Mac.

I would use it with FreeNAS as well. Just put FreeNAS on a flash drive (128+ CF Card connected to the standard IDE), and use all drives as your data drives only.

FreeNAS has another feature which might be useful for you, that's ZFS. It's still experimental so you may want to run a several tests to make sure you understand all its tricks.
http://en.wikipedia.org/wiki/Zfs

 

[HappyBison]

P.S.: I use FreeNAS with Windows and Mac using SMB protocol.

 

[ldyt]

It is subject to RAID controller card that how many Hard Disks will the server connect.
I suggest that you buy a RAID controller card, such as dell perc 5i, lsi 3081e for 8 HDD, adaptec 4805 for 12HDD with sas cable.

If you have other question, pls email me: aiai-huang@hotmail.com

 

[rand_79]

there is a reason the orginal solution was 2500$ going cheap sometimes isnt good.

 

[ldyt]

USD20 for 10pcs sata cable, i don't think it is cheap, it cost 0.3/pcs at my city.
We often can buy some raid controller card and fibre card at best price.

 

[ldyt]

Sorry, our city is a bigger international electronic market, we often consider the price and quality at the same time.

 

[sub mesa]

If you go FreeNAS or any other non-windows OS, you don't need a RAID controller. That's the major advantage here. So you can build a cheaper system and have generally more value for your money.

A self-made solution does require you to spend some time on it though, including a testing period once you get everything up working. This is very important, if it'll used for production purposes.

If you want to connect more SATA disks, this SATA non-RAID controller is a good pick to add room for 8 additional HDDs. This can enable you to choose cheaper 1.5TB models instead, or simply have room for a bigger array. Find the controller here:
http://www.supermicro.com/products/accessories/addon/AOC-SASLP-MV8.cfm

You can use these for any software RAIDs or filesystems, so you're not tied to using the same controller you can use both onboard SATA & addon SATA controllers ports. FreeNAS will work fine with either Linux, Windows or Mac; though its an OS on its own and needs to be installed to a computer dedicated to storage. So the system you install FreeNAS too isn't a system you can do other stuff on, except if you put FreeNAS in a virtual machine, such as with the free Virtualbox package.

As far as RAID goes, you have the choice between RAID0+1 or RAID1+0, RAID3 and RAID5. If you are going to use the ZFS filesystem, it has its own RAID implementation and can do RAID5 ("RAID-Z") and RAID6 ("RAID-Z2") very well. But its experimental in FreeNAS, while OpenSolaris and FreeBSD 8 have more stable implementation of ZFS.

I guess you should decide:
1) NAS or DAS? In other words, is the fileserver going to be dedicated or do you need to work on it
2) RAID/Filesystem (ZFS, geom RAID available in FreeNAS paired with UFS, journaling? etc)
3) number of disks, controllers needed, components for the PC
4) how to setup/configure properly

 

[sminlal]

A self-made solution does require you to spend some time on it though, including a testing period once you get everything up working. This is very important, if it'll used for production purposes.

Sub Mesa is 100% on the money with this. If you're going to use a RAID system that provides redundancy for your data, I can't stress enough how important it is to test your recovery procedures before you commit live data to it. Pull a drive out of the array and see what kinds of notifications you get. Put the drive back in and document exactly what steps it takes to rebuild the array so that it's "healthy" again.

I've seen tales in this forum of people who thought their data was safe on a RAID system but ended up loosing it all because they didn't understand exactly what was required to get the array rebuilt. All RAID solutions (hardware or software) have their own little quirks and it's important to have a good understanding of them if you're data is important.

And of course, don't forget that even a redundant RAID set needs to be backed up to protect your data against all of the other risks that RAID can't protect against.

 

[Bruceification73]

"2 GB of RAM is OK, right? What about 4 GB for an extra $30?"

Go with the 4 GB.

 

[MRFS]

The comment above about the SATA ports on the P5Q Premium is correct:
the orange and blue ports are special items that have caused some
difficulty for ASUS users. They're called "Drive Xpert Technology"
using a Silicon Image SIL5723 controller that could be better documented:

the orange pair are wired to one such controller, and the
blue pair are wired to a second such controller.

We have a P5Q Premium and we speak from direct experience.


BEST WAY is to invest in a solid hardware RAID controller with
plenty of SATA/3G ports. We've had a lot of excellent results
with the Highpoint RocketRAID 2340:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816115031&cm_re=N82E16816115031-_-16-115-031-_-Product


Highpoint has much newer and better RAID controllers now:

http://www.highpoint-tech.com/

Even if you go with a 32-bit OS, Highpoint's device drivers
permit variable sector sizing, in order to overcome the 2TB
limitation when 512-byte sectors are the default.


Just be sure that you select a controller with an x8 edge connector
and that your motherboard's chipset can assign x8 PCI-Express lanes
to that expansion card: you don't want the chipset to assign x4 lanes,
if you can avoid it.

Also, for that large number of HDDs, I would also highly recommend RAID 6
because of the added data security it buys you, for only marginal impacts
on efficiency as compared to hardware RAID 5.

You definitely want hardware RAID logic because these redundant
arrays -- RAID 5 and 6 -- do a lot of parity computations.


To "future-proof" your RAID controller choice, take a look
at Intel's RS2BL080 and RS2BL040 SAS/6G controllers:

http://www.intel.com/Products/Server/RAID-controllers/RS2BL080/RS2BL080-overview.htm
http://www.intel.com/Products/Server/RAID-controllers/RS2BL040/RS2BL040-overview.htm

The future is 6G and, later on, solid-state 6G e.g. SATA/6G and SAS/6G.


The rest is easy.


MRFS

 

[MRFS]

p.s. for a general-purpose file server, the E3200 will save you some money:
it still has the Wolfdale core (just ignore the "Celeron" re-branding):

http://www.newegg.com/Product/Product.aspx?Item=N82E16819116265&Tpk=E3200


MRFS

 

[MRFS]

> This motherboard should be able to take all 10 hard drives and make them into one disk with a RAID 5, right?

Definitely NO! (see above)


Repeating: invest in a quality hardware RAID controller
e.g. LSI, Areca, Intel etc. Highpoint may also have a
controller that fits your needs for the foreseeable future.

You can reach 16TB with 8 @ 2TB, e.g. Western Digital's WD2003FYYS :

http://www.wdc.com/en/products/products.asp?driveid=732 (highly recommended)

The latter 2TB HDD is a really beautiful enterprise-class HDD:
we have 2 of the RE3 750GB version and in RAID-0 they smoke tires:

http://www.wdc.com/en/products/products.asp?driveid=504
http://www.newegg.com/Product/Product.aspx?Item=N82E16822136316&Tpk=WD7502ABYS


That should simplify your controller choices.



MRFS

 

[endorphines]

Intel chipsets do not allow arrays of over 2tb. i know everyone is going to say "no that's your os" but it's not. I've tried on 4 different chipsets and even in the bios utility the largest array they allow is 2tb.

my advice is either a massive nas. 8 port would be your bear minumum. with 2tb drives, that'd give you 14tb with a raid 5 array. with that kind of capacity you need redundancy because the chances of loosing a drive when you have that many is kindof high. you could always pick up an enterprise level raid card and put it in a server. i have a highpoint rocket raid 2320, which is an old model now that's been replaced, but it had a feature, where if you have two, you hook them together and you can run your 2 8 port raid adapters as if they were a single 16 port.
if you want to build this thing yourself, that's the rout i'd take, as migrating that system once it's full would be quite the task. this way you can use the online capacity expansion feature. and just add 6 or 8 more 2tb drives once they're cheaper.

 

[sub mesa]

If you use Software RAID on for example FreeNAS, such limits do not apply. The hardware will never know your storage array is larger than 2TiB; it will only see that multiple disks are accessed, with each disk being 2TiB or lower capacity; so never will 64-bit LBA be required; only by the operating system itself which is pure software.

A solution this size begs for something like ZFS, by the way.
Should also be much more reliable than a 'conservative' filesystem + RAID solution.