Sign in with
Sign up | Sign in
Your question

Building 7TB RAID5

Last response: in Storage
Share
April 22, 2009 11:05:37 AM

I am considering picking up:

(8) 1TB drives and a RAID5 supporting card to build a file server for my house.

I have spent hours on newegg looking at highpoint, rosewill, adaptec, LSI and the like and have some questions.

1. There are a few cards with 2 mini-sas connectors. Would I be able to put 4 SATA drives on each and merge them together into 1 RAID configuration?

2. Would you recommend 1 card for this setup or 2?

3. Do you have any recommendations for a card? I saw this one and thought it might work:
http://www.newegg.com/Product/Product.aspx?Item=N82E168...

4. Any other thoughts or recommendations on drive make/model are greatly appreciated.
- I've considered the SAMSUNG drive: http://www.newegg.com/Product/Product.aspx?Item=N82E168...

5. To handle non-gaming OS use and the RAID setup, what kind of power supply should I be looking at?

6. I don't have any of the hardware yet, but would like to find the right balance of performance and energy efficiency. Are there articles that address building servers of this type for home use, or do you have recommendations on boards, cases, et al?

Thanks in advance!
-Chris

More about : building 7tb raid5

a c 126 G Storage
April 22, 2009 11:23:43 AM

If you're building a fileserver, why not pick an OS suited to that? If you would run FreeNAS/FreeBSD/Linux you could save the cost of a hardware RAID controller and go with software RAID instead

Should you go for the controller, then yes the mini-sas just connects multiple SATA you can make one big RAID5 if you want.

5400rpm disks are to be recommended, so look around at WD Green's portfolio, the WD10EADS is a very good and power efficient drive. Meaning you won't have a huge build-up of heat near your harddrives, which is a good thing.

You need a power supply that can boot your system, meaning <150W> static + 35W per drive spin-up current. With 8 disks that makes 430W. Investing in a good power supply is important when building a NAS, since your RAID will mean nothing if a power failure in your PSU causes all your harddrives to fail because of improper voltage. That's not theoretical it happens to ppl. :) 

My advice: think about your OS first. It'll decide for you if you need a controller for the RAID; since Windows doesn't offer any advanced RAID functionality. However, with software RAID you could use the advanced ZFS filesystem. Howmuch experience do you have with NAS and storage, what are your demands?
April 22, 2009 12:20:51 PM

Thanks for your response. I should have been more specific. I'd like to have a setup where I install VMWare ESXi server, allowing for me to have a couple hosts running (a Windows, Ubuntu), and then they both could leverage the non-host providing portion of the storage.

So in addition to being a simple file server/storage, the hosts will allow me to use the server to do encoding, host a web server etc..

This being in a home, I would probably have some minimal streaming (music, videos), and have the web server.

There were two routes I was considering taking:

1. A VMWARE ESXi server setup with the following hosts:
a. Windows-type host (general windows computing needs when need be)
b. Ubuntu host (providing a linux environment)
c. FreeNAS (exposing the rest of the storage as a NAS?)

2. A VMWARE ESXi server setup, and a RAID5 card setup:
a. Windows-type host (general windows computing needs when need be)
b. Ubuntu host (providing a linux environment)

What do you think would be the proper setup given my needs and would I need a RAID card given that setup?

I have no experience with NAS or storage, other than leveraging it at work. But I'm capable of reading directions, searching google for answers, and leveraging forums.

I'm a software engineer by trade, so I'm someone capable :) 

Thanks,
-Chris
a c 126 G Storage
April 22, 2009 1:14:22 PM

The setup you explained sounds good on paper, but how reliable/good is the VMware solution? I have no experience with those products, though if i understand its concept it wouldn't allow you any 3D acceleration on the virtualized operating systems. I don't see any word like "video" or "3d" on the advertized features:
http://www.vmware.com/products/esxi/features.html

So while it may work, you're stuck without 3D and if the vmware solution has issues or problems it may not prove a very resilient setup. If the VMware part fails, all other three do as well.

I'm not sure how much learning you're willing to take. But what if would propose to a different setup, with no vmware but rather a normal ubuntu desktop installation, and then using virtualbox (free vm) to host both windows (you can try out different versions like XP, Vista, Win7) and your storage/webserver. So you would have:


X Ubuntu 9.04 amd64 (main operating system)
- Windows XP 32-bit (guest OS)
- Windows 7 64-bit (guest OS)
- FreeNAS/FreeBSD server (guest OS)

This setup could provide you with a normal OS (plain ubuntu) running Virtualbox to virtualize other operating systems like Windows. This would allow 3D acceleration for both Ubuntu and Windows (though Virtualbox only supports OpenGL at the moment - DirectX coming in next releases). Just make sure you have enough RAM in the system (8GB or more should do) and you pick a 64-bit distribution of Ubuntu you're fine.

The Ubuntu host OS could also do the webserver part, though for maximum security you can do this in a VM instead. The Ubuntu host OS could also do the NAS/storage part, and it would allow you to use ZFS via the FUSE interface. Another option would be FreeNAS/FreeBSD in a VM.

This setup would also allow you to get familiar with Ubuntu and Linux in general, you can try out which applications suit your demands and which not. Wine (the program allowing you to run Windows-executables on Linux et al.) can run many applications today, and those it cannot run you can use in the Windows guest. This setup would be preferable if you're keen on expanding your horizon into the linux world, it would allow you a good opportunity to get to learn alot while still having a "fall-back" to old good windows as a guest OS. So, are you ready to embrace Tux ? ;-)
April 22, 2009 3:29:56 PM

Sub,

I'm not too opposed to learning new stuff and work with redhat at work for all deployments of code.

Given your setup here are my questions:
1. In your setup, would the motherboard be supplying the RAID? Or would FreeNAS provide the Raid?

2. If in your setup, there is not a controller based RAID, would I install Ubuntu on a separate drive from the RAID collection of drives, then RAID all the HDD via FreeNAS?

I'm just curious where the OS stuff would be installed. Would you put it on the RAIDed drives, or would you have a separate drive for OS related stuff and then the (8) 1TB drives together comprise the 7TB Raid5 would be used for the RAID?

Thanks for your continued assistance!
a c 126 G Storage
April 22, 2009 5:21:10 PM

You can use partitions, meaning one small partition for the OS and a bigger partition at the end to be used for the RAID-array. But that would mean all disks would shrink to an equal amount of the first partition, which is probably not what you want.

The prefered choice is to use dedicated disks. So if you budget allows i would get an SSD drive as main system disk (Intel X25-M would be best, or a Vertex as value option) to install all your system stuff on, improving the general responsiveness of the system. This would allow you to boot without requiring any RAID drivers or special configuration -- the least cumbersome and problem-prone method. The SSD could have multiple partitions with a "backup" XP install outside of virtualbox, should you ever need that. But the important thing is, you have the data disks fully dedicated to the RAID-array now, so it simplifies your setup, which is good.

The RAID could be done either by the host OS (ubuntu - yes even the desktop edition that really doesnt make any difference in this regard - just install mdraid via synaptic and you're good to go) which could provide maximum efficiency. However, for both ZFS and Raid5, more advanced software is available in the FreeBSD operating system. FreeNAS is also integrating these two technologies, the raid5 driver is already present. However, i would propose you would use FreeBSD instead (which is what FreeNAS is built on). I could also personally help you setup that part.

Virtualbox would have to be configured by allowing FreeBSD physical access to the data harddrives (not your SSD though). The FreeBSD operating itself will just be a fixed image of say 6GB on your system drive. Once configured, you can make a "snapshot" of the VM essentially creating a backup. If it ever stops working in the future, simply reverting to the backup should clear things up, unless its a hardware/persistent issue ofcourse. It would also allow you to make mistakes when trying to learn things. Just make the snapshots before you do anything dangerous. The FreeBSD VM would have samba or NFS or SSH to serve the actual files. Going this route you will have access to kernel-level implementation of ZFS (Zettabyte Filesystem - read up on it on wikipedia) and also a high-speed RAID5 driver - which writes faster than what linux can do.

Either way (ubuntu doing the RAID or FreeNAS/FreeBSD doing the RAID) its both software RAID - not onboard RAID which uses proprietary Windows-only drivers to do the actual RAID work. In this case the native implementation of Linux and/or FreeBSD is used.
a c 126 G Storage
April 22, 2009 5:26:02 PM

I also would like to add, using such a setup means true independency from specific hardware. You could clone all the data disks to completely different disks and build a completely different system with different operating system. As long as it runs Virtualbox fine it should work regardless of any changed hardware. So you're more flexible in this regard. You could migrate the disks to another system and run the VM there or any of that - should you want to do so in the future.

Ofcourse it does make you dependent on the Virtualbox software, but it works pretty good to my experience, and development of it is ongoing with exciting new features (like 3D support) being added every few months.
April 22, 2009 9:14:12 PM

Depending on what type of material you are trying to store in your raid array, an inexpensive solution to take a look at is called Unraid, made by http://lime-technology.com/. I have a professional license running 13 hard drives at the moment. It is Linux-Slackware based, and have heard people modding the stripped down version to run VMWARE as well as do a ton of other things. This technology is not a true RAID technology but does provide some redundancy for your data, does not require a hardware based RAID card, can use both IDE and SATA drives of differing sizes. With a caching drive installed, write speeds are not too bad and since it only spins up the drives the material you need is on, it is fairly energy efficient. For those 13 hard drives I have a 500 watt power supply and it seems to be running fine. Good luck on finding your storage solution.
April 22, 2009 10:16:26 PM

Hey Sub,

All of that sounds great, and I think for my needs I will be able to go with RAID software solution, either with ubuntu, or motherboard provided.

Just to provide myself with the most flexibility, I'm trying to pick the best MoBo to support raid solutions and have enough connectors to support at least 8 Sata drives.

Do you have any recommendations in that area?

I tried to find some 5400 1TB drives as well, on Newegg.com, but they don't have any available. Should I just shoot for the 7200 RPM?

Thanks again
April 23, 2009 9:02:38 AM

Why not go for fewer seagate 1.5tb drives or just 4 2tb wd greens?

ps. wd claims those greens aren't good in raid, as their way of error handling takes too long for hardware controllers. And the controllers would drop a drive out of raid even if it isn't actually broken.
I'm not sure, but I believe the firmware in most of seagates 1.0 and 1.5tb drives have the same problem - on top of the december issue where drives just died that is.

Link to info on the WD issue: http://wdc.custhelp.com/cgi-bin/wdc.cfg/php/enduser/std...


Anyway. I don't know enough about the controllers to give answers, but I'd recommend going for a drive that is raid certified. ie. WD RE3 drives or equivalent from other manufacturers.
April 23, 2009 9:34:25 AM

None of the Intel motherboards have more than 6 Raid ready Sata ports and if you want onboard Raid, Intel is certainly the best.

But I think the bigger picture is being missed: software Raid 5 and even hardware Raid 5 is horribly slow when you add writes to the mix. If it was just a file and web server, Raid 5 would work great. But since you want VMs and encoding, a single Raid 5 array, whether software or hardware will slow it down. A good reason to get a hardware controller, and that Highpoint 4320 is an amazing value, is hardware controllers use delayed spin-up; thus, the power requirement during boot-up is greatly reduced.

Sub mesa knows quite a bit and it seems that you would need to spend weeks upon weeks learning to implement what he suggested. I prefer not to trust my data with free software and would rather stick with the big guys(VMWare) and hardware.

I forgot to add: using the Highpoint and 8 drives, 1 -Raid 10 & 1 -Raid 5 or 2 -Raid 10 arrays would provide far better performance. You could put the File server on the Raid 5 and the VMs on the Raid 10.

For drives, the new 7200.12s are the fastest and lowest power 7200rpm drives ever.
a c 126 G Storage
April 23, 2009 12:57:46 PM

Define 'slow', is 400MB/s slow? With write-back the writing part has some tricks in order to hide the complicated writing process on RAID5. Software RAID5 is known to beat Areca in sequential read and write scenario's, but that's not so hard; the Areca is still superior for anything but sequential I/O.

For ZFS, the RAID-level used is called RAID-Z. It is comparable to RAID5 but differs in the way that dynamic stripe sizes are used; unlike RAID5 where a static and unchangeable stripesize is used. By having variable stripes, many of the 2-phase write techniques required by RAID5 are not required by ZFS's RAID-Z. This is only possible if filesystem and RAID-engine are integrated in one package, like ZFS basically is; filesystem and RAID-engine. Its pretty great, and ZFS is so superior to well-known filesystems like NTFS and Ext3.

By the way, Virtualbox can export/import images to standardized formats, so you could use the VM probably with other virtualization products like vmware as well; so you're not stuck with one vendor.

One nifty feature of hardware controllers specialk90 pointed out is staggered spin-up, which means the disks don't try to spin-up all at once meaning you can use a less powerful (but more efficient) power supply to power your disks.

Chipsets have maximal 6 SATA ports. You can extend them with PCIe controllers like Silicon Image SiI-3132, JMicron JMC363 or other cheap controllers which transfer via PCI-express x1 port. These work in both linux and bsd and you can use them to add more ports to your system. They run at full speed, though 2 channels are limited to 250MB/s full duplex which is the bandwidth of the PCIe x1 slot. They should be cheap and you should be able to use more of them in one system; but they do not support staggered spin-up so watch your PSU if you go this route!

http://www.made-in-china.com/image/2f0j00NvEaCKRYvgkQM/...
example of simple add-on controller, often offering two additional SATA/300 ports

If you want the RAID-edition drives, you can checkout the WD Green 2TB RE4-GP disks, dunno their price but compare all your options and go for something you can use for a long time and have alot of fun with.
April 23, 2009 7:46:51 PM

Sub mesa, could you help me out? I use Adobe CS4 for my job so I use Vista x64. Is there a way to use ZFS for my Raided drives (4-500GB in Raid 5 currently and possibly 4-150GB Raptors in Raid 0 or 10, in addition to the 4-74GB in 10 & 0 for OS/Apps using ICH) and what is the benefit to ZFS vs NTFS? Would ZFS prevent partition corruption? I never want to go through that again, although I wouldn't need to spend 40hrs trying to find a program that can save my data.

Thanks
a c 126 G Storage
April 23, 2009 8:02:00 PM

Windows cannot support ZFS directly, but if you would run Windows from within a VM you could store the image files on ZFS.

ZFS's design is much different from other filesystems. Its the first widespread filesystem with its own RAID-engine and it uses several new techniques which are fundamental in nature, such as the Copy-on-Write model. In essence, ZFS should be one of the most safe filesystems:

- checksums protect both the data as well as the metadata (metadata is 'extra' data used by the filesystem, if it becomes corrupt on NTFS, you can loose whole terabytes worth of directories. it may be recoverable but hey, preventing that in the first case is better)
- because corruption is now detectable, ZFS can correct/fix/heal this on-the-fly, as long as its a correctable error. Should you have a RAID1 or RAID5 array, ZFS can use the redundant data to 'correct' the corruption, or simply restore the file from a backup or snapshot copy.
- no more checking filesystems; ZFS is maintenance-free! ZFS heals itself, whenever it feels the need to, you can force a full 'scrub' which will read all files and if the "checksum error" counter stays on zero, your filesystem is in perfect health.
- on the fly compression and AES-encryption (zfs-crypt is a feature in development; not yet integrated in most dists)
- Since its copy-on-write, should the system crash or a power failure presents itself, the filesystem will simply be in state in the future (like 20-30 seconds), but it will be consistent so corruption should be no issue in this case
- ZFS can be very fast, in the right circumstances, the Linux ZFS implementation is not so fast because of license issues, but on FreeBSD and OpenSolaris you can get the latest version offering great performance and features, they are both kernel-level implementations.

So if you're asking if ZFS is safe, then the answer is yes. By design it has many more safeguards than say NTFS (which only has weak metadata-only journaling) which should make it safer. There is always the possibility of a bug in ZFS though, although many people are running it without too much trouble. The biggest issue so far is that ZFS needs loads of memory and a 64-bit distribution is more or less required. Look for more info: http://en.wikipedia.org/wiki/ZFS
April 24, 2009 10:40:35 AM

Sub,

So I've come full circle and are very excited about trying the OpenSolaris route to leverage ZFS.

It's the architecture of the system that I'm still uneasy about.

1. PCI-E 4-port SATA adapters that appear on the OpenSolaris HCL are either really expensive (probably) or non-existent (my current observation).

2. Would I be able to get around the hardware compatibility if I exposed all of the HDD to OpenSolaris via VMWare, and then inside an OpenSolaris VMWare host create the RAID-Z pool?

Or let's put it another way.

If you had 8 SATAII 1TB drives (+1 for the system drive), and you wanted to use ZFS but also provide for virtualization of other operating systems, how would you implement?

Thanks again!
April 24, 2009 10:47:01 AM

Sub,

I'll just add ..

You mentioned in an early thread in this post the following configuration:

X Ubuntu 9.04 amd64 (main operating system)
- Windows XP 32-bit (guest OS)
- Windows 7 64-bit (guest OS)
- FreeNAS/FreeBSD server (guest OS)

Would OpenSolaris play in that mix, being the ZFS provider instead of FreeNAS/FreeBSD?

Thanks
April 24, 2009 10:55:35 AM

And additionally, if the hardware compatibility is much better for Ubuntu, and the ZFS configuration and reliability is at least the same as OpenSolaris, then I would strongly consider just using Ubuntu.

Can you speak to that as well?
a c 126 G Storage
April 24, 2009 1:27:40 PM

OpenSolaris is a different beast than FreeBSD, but since all other operating systems run inside a VM; the hardware compatibility is not important here. Virtualbox will use its own SATA-controller to advertise the drives to the virtualized operating systems (FreeBSD or OpenSolaris). So only the host OS (ubuntu) would need to support it, while FreeBSD or OpenSolaris are actually using it. So it would be like:

SSD/small disk: contains host operating system, and disk images of the crucial FreeBSD/OpenSolaris VM. Once you boot that, it will connect to your RAID-array harddrives and make the huge filesystem available to the host operating system either via "shared folders" (never used this) or via normal netwerk access (samba/ssh/nfs). You can then mount the RAID-array in your Ubuntu host OS; ubuntu wouldn't know its RAID; it also wouldn't know its ZFS; it just sees a large storage space. All the managing of the array you can do inside the VM. For example, if you want to add a filesystem (you can do that on-the-fly without any configuration) to the pool, or enable compression for a certain directory, or check the status of the pool... all you need to do is click virtualbox and work with the virtualized OS. Hope that much is clear to you. The question really is:

- how reliable is a virtualized OS
- do bugs persist in virtualbox that might spoil your fun
- how much performance would such a setup deliver, where the RAID is done by the VM but the host mounts the RAID storage by interfacing with the VM with Samba/NFS/SSH

I'm trying to recreate a FreeBSD VM environment which has access to 6 physical disks. I'm setting that up now with Virtualbox, to see for myself what kind of performance you would be looking at, and if it would work (well why wouldn't it).

But the good parts:
- hardware compatibility is no real issue anymore if you use them inside a VM
- as long as your host OS (ubuntu) supports your hardware, your guests will also. Virtualbox uses its own SATA-controller for the guests. The guests wont even see the disks aren't all on the same controller.

As for OpenSolaris vs. FreeBSD. I have experience with BSD but not with OpenSolaris. So while i cannot recommend you OpenSolaris myself, its a good OS with nice technology. Yet, there's no real advantage using OpenSolaris vs FreeBSD 8 regarding ZFS, since both use the same implementation (same feature level). Is there any compelling reason you want to use OpenSolaris? It has the same (memory) restrictions the ZFS port in FreeBSD 8 has.

The cheap PCI-express controllers costed me 11 euro each, so they should be really cheap; the problem is finding one. They are not sold by their controller-names (like JMicron/Silicon Image) but rather by third-party packagers. They will be supported in Ubuntu and BSD, but i'm not not sure about OpenSolaris. They will generally be no-name SATA controllers with PCIe x1 interface and just 2 or maximum 4 SATA-ports. It may also be advertized as a RAID-controller; though that's just when you install the windows driver. Hope you can find it though.

Since FreeBSD 8 offers the exact same ZFS implementation with all relevant features, and Sun has recently been acquired, i personally would opt with a known free operating system. It would also allow me to help you more thoroughly should you choose to start implementing the idea. Since storage is something i love, i wouldn't mind helping people with exotic storage plans at all, if i can. :) 
a c 126 G Storage
April 24, 2009 1:30:07 PM

Something like this: a simple PCIe SATA controller. This one has only one SATA but you can see it holds space for two ports; those are the regular types. Don't buy a PCI controller it has to be PCI-express for performance reasons.

!