Sign in with
Sign up | Sign in
Your question

ZFS vs hardware RAID for low power file server

Last response: in Storage
Share
November 12, 2009 10:52:27 AM

Hi All,

I asked a while back about creating an ultra-low-power file server for a home network. I hadn't considered using ZFS at that time - was looking at a dedicated RAID controller. I'm considering an Intel Atom or Via Nano based system, probably running FreeNAS. I guess a software-based system will tax the processor more... Can anyone comment on the pros and cons of using a dedicated controller, particularly with respect to power consumption and performance?

Ta - Adam...
a c 127 G Storage
November 12, 2009 11:22:50 AM

You may want to read through this thread:
http://www.tomshardware.co.uk/forum/page-253172_14_0.ht...

Don't use hardware RAID if you want to use ZFS - you'll lose one of the most important benefits of ZFS - the self healing process. Software RAID in these cases is clearly superior to hardware RAID.
a c 127 G Storage
November 12, 2009 11:23:55 AM

By the way, Intel Atom is 32-bit right? You should use 64-bit CPUs if you want to use ZFS. Low-power AMD based systems use less power than Intel Atom based systems, as these often come with old chipsets that nullify any power benefit the Atom cpu provides - Intel has to dump its old stocks somewhere.
Related resources
November 13, 2009 8:37:06 AM

Hi sub mesa,

Thanks for all the helpful info. There's loads to get through...!

I'll make sure I only consider 64-bit architectures. I assume there's a trade-off between CPU power consumption and controller power consumption when considering ZFS vs hardware driven RAID. IOW, I'm assuming that:
- ZFS solution = gruntier CPU = more watts, but controller consumes less power;
- Hardware RAID = weedy CPU = less watts, but controller consumes more power;

Is that right? Do you know which will be more efficient in terms of power consumption?

Will a chip like the Via Nano be heavy-duty enough to run, say, FreeNAS and ZFS or OpenSolaris and ZFS?

Cheers - Adam...
November 13, 2009 1:15:33 PM

some of the WHS machines available are very low power and very robust. Mine is a WHS frankenmachine and it has never let me down for backups and storage.
a c 127 G Storage
November 13, 2009 1:24:55 PM

Low-power means something like 35-50W excluding disks. This is easily achieveable with an AMD setup and efficient power supply.

For example:
  • AMD Dualcore with 35W or 45W tdp means ~2-3W idle
  • AMD 740G/780G/785G chipset means ~2W idle
  • Amacrox 400W fanless PSU is 90%+ efficient, one of the most power efficient power supplies and comes without a fan

    If you go for 2,5" HDDs you can even go for an "extreme" setup with PicoPSU power supply, which is like ~97% efficient and can reduce idle power consumption to 30-35W, while still providing high performance with lots of memory and dualcore chip to saturate your gigabit NIC.

    Note that TDP can be misleading; while the maximum power consumption is important for chosing the correct power supply, its not important for actual power consumption, as your hardware will idle most of the time. In fact, all PCs in the world idle more than 90% of the time. Idle power consumption is what you should look at if you are concerned with the environment and/or want to save on energy costs.
    November 13, 2009 8:11:08 PM

    Having 64-bit for ZFS is good because ZFS can use your memory really good so it make sense to max it up with 4GB of RAM or more (32-bit architecture usually can't access more than 3GB or so).

    Back to your topic.
    I would put ZFS with RAID-Z (AFAIK, there are no hardware RAID-Z controllers so it will be software RAID).

    Regarding the power usage, I built my own ZFS NAS using FreeNAS using Intel Core2 Mobile.
    If you want to get Intel Atom, they have Atom 230 and Atom 330. Both of them are 64-bit. Atom 230 has one core, Atom 330 has two cores. Atom 330 has 8W TDP.
    a c 127 G Storage
    November 13, 2009 10:21:20 PM

    There are more reasons why you would want 64-bit CPU for ZFS. This advanced filesystem is simply not stable on 32-bit systems, due to a maximum of 1GB kernel memory. In many cases with heavy disk I/O, ZFS would cause the operating system to crash with a "panic: kmem_map too small" message. Its possible to tune ZFS heavily so it runs reasonably stable on 32-bit, but that would mean disabling lots of stuff that decrease performance.

    32-bit is limited to 2GB per process; its Windows that uses up to 3GB and 1GB kernel memory for 4GB memory systems. With PAE you can have like 64GB RAM on 32-bit systems, but the 2GB-per-process still applies, unless the process/application is specially compiled which is extremely rare. Performance with PAE would be affected too. You really want to have a 64-bit system in this case.

    @HappyBison, what kind of ZFS setup do you run and are you happy with the setup?
    November 13, 2009 10:47:16 PM

    sub mesa,
    I'm still experimenting (not using it for real life data), but I'm happy with what I'm getting.
    I use FreeNAS (ZFS v6) and FreeBSD (ZFS v13). I downloaded OpenSol just to compare and benchmarking, but I'm more FreeBSD guy.
    a c 127 G Storage
    November 13, 2009 11:07:38 PM

    Good to hear others are running FreeBSD as well. Are you running 7-STABLE or 8.0?

    I haven't compared FreeBSD ZFS performance directly to OpenSolaris yet, if you would have any benches i would be very interested in them. :) 
    November 14, 2009 12:05:37 AM

    sub mesa,
    I have FreeBSD on all of my online servers (7.0 and 7.2). I prefer not to run Beta and RC on production boxes :-)

    As far as the benches, my goal is to measure it in all aspects I could and post it online (I could not find any info on that so I'm building my own when time permits). I bought different kinds of hardware (Core2 Mobile, Atom, etc) to see how that will affect the tests too.
    a c 127 G Storage
    April 17, 2010 2:05:27 PM

    Uses more power than AMD/Intel based setups; due to their chipset (VX800 if i recall correctly). In fact, with an AMD setup you go as low as 5W idle; though that is extreme and requires a custom power supply (PicoPSU).

    With a regular low-power system you should be targeting 45W at the wall socket, without disks.
    April 18, 2010 7:23:03 AM

    sub mesa said:
    Uses more power than AMD/Intel based setups; due to their chipset (VX800 if i recall correctly). In fact, with an AMD setup you go as low as 5W idle; though that is extreme and requires a custom power supply (PicoPSU).

    With a regular low-power system you should be targeting 45W at the wall socket, without disks.


    Hmm,... I am also looking for a low power server and had just discovered this VIA MServ S2100 as an option for myself, haven't found a source to buy one yet, maybe they're not being sold yet.

    I'm a newbie when it comes to servers. 5W idle sounds great, but I need one that can handle a few things, such as VMware virtualization, with a virtual server for network services (LDAP, RADIUS), a virtual database server (Oracle) and an instance of Windows 7 to use as a workstation at my desk. There won't be much load on any of these and likely a lot of idle time. The configuration you imagine, with the "AMD/Intel based setup", reckon it could handle this? I'd be very interested in your shopping list, if you have already compiled one :)  .
    a c 127 G Storage
    April 18, 2010 11:55:17 AM

    5W idle at the wall socket? I do not believe, with the VX800 chipset. Perhaps the CPU alone. I would try to gather more detailed information, if you are going this route.

    For all intents, you may just be sacrifising performance compared to an AMD/Intel setup, while not saving any power. It is like the Atom platform with Intel's old chipset stocks which intel wants to get rid of. That means you get a 2W CPU and combine that with a 945GC chipset that uses 22W; so the chipset uses more power than 11 Atom CPUs together at maximum power. Nice energy efficient platform!

    If you want to go Atom, the new SoaC-chip which includes the chipset in the CPU die is a nice option i think. But it's very low performance, for ZFS you want a real CPU unless you're okay with low performance. Also remember that you need a 64-bit CPU.

    The simplest would probably be an AMD or Intel solution on Mini-ITX or Micro-ATX platform; those are getting more popular by the day and are also quite power efficient, while packing a lot of muscle to do hard work when needed.
    April 21, 2010 10:59:17 AM

    Hi,

    I was wondering if I could hijack this thread a little, I'm having similar questions regarding home NAS, ZFS en software raid (not hw raid).

    The goal is to set up a home NAS server with a few initial disks (say 6 or 8 t)b, and over time expand it to 15-20 tb. The growing really is an important factor, I intend to use this machine for a few years.

    I've thought long and hard about software raid / hardware raid / os / zfs / raid 5, 6 / hardware / ... I think I've now gotten to a point where I think I know what I want/need, but I'd like some opinions / sugestions to change / improve the setup.

    I'd go for a DIY debian (lenny) server, using a software raid 6 array (with mdadm).
    Because:
    - Linux because I'm familiar with it, not the big expert but I get around using it
    - MDADM is a great implementation for software raid, very flexible and supports both growing raid5/6 arrays, and adding hot spare disks
    - No ZFS because I would really like the opportunity to grow the array, and ZFS (at the moment) doesn't support this. If some1 can tell me that this will be implemented next year or ... then I probably would change a lot of things in this setup
    - Raid 6 because I've read too much stories about Raid 5 failing or corrupting data when rebuilding from standard desktop hard drives
    - Ecc memory because it would generally support data integrity

    Here's what I'm thinking about regarding hardware:
    CPU: AMD Athlon II X2 240e
    Pretty easy, a low power 64bit CPU

    Mobo: ASUS M4A89GTD PRO
    I've picked this because of the onboard video and the 2x PCI-2 8x ports, this would allow me to (in the future) add a lot more sata disks through interface cards

    Memory: 2x Kingston 2GB 1066 Mhz DDR3 ECC
    Doesn't need any more info I think

    OS HD: Western Digital Scorpio Black WD1600BEKT
    A 2,5' disk for the OS, I think the system would benefit from using a 2,5 instead of a 3,5 regarding power consumtion. Since the OS disk never spins down (or hardly ever), this does require less power. Not sure if I should considder changing this to a 7200 rpm model.

    Raid: 5x WD Caviar GreenPower WD20EARS, 2TB
    I was fairly sure of this, but I'm thinking of changing to the F3 model from samsung

    HotSw: 3x Addonics AE5RCS35NSA
    To neatly organise 15 raid hard drives, with hot swap

    PSU: <tbd>
    Not sure which one yet. Since It has to be able to run a system with 15+1 hard disks I think I need something with some power, but I'd like it to be as efficient as possible.

    Case: Cooler Master Cosmos S
    10x 3,5 with front access, 9 will be used by 3x hotswap bay, one remains to add a DVD I guess

    Any comments and suggestions regarding software and/or hardware setup are very welcome :-)
    a c 127 G Storage
    April 21, 2010 1:39:38 PM

    With ZFS you actually can expand, if you add 4 disks at a time. Does that change your perception on ZFS?

    The EARS drives would be better suited to cope with BER or Bit-Error Rate. That's a problem with many disks in RAID5 or RAID6 without using a filesystem that uses checksums like ZFS/Btrfs.
    April 21, 2010 1:50:43 PM

    sub mesa said:
    With ZFS you actually can expand, if you add 4 disks at a time. Does that change your perception on ZFS?

    The EARS drives would be better suited to cope with BER or Bit-Error Rate. That's a problem with many disks in RAID5 or RAID6 without using a filesystem that uses checksums like ZFS/Btrfs.


    Thank you for your reply.
    It would actually, if this didn't mean that I lost 2 out of the 4 disks on parity, which I think will happen.

    With mdadm you can just grow you array and keep the same number of parity disks, but I don't think that's possible with ZFS at the moment, is it ?

    I mean, I'll start off with 6 disks, 1 OS and 5 raid 6 = 6 TB, lets say in 6 months I'll add another 5, so now i'm at 16 TB with mdadm, where I'd be at 12 with ZFS. At least that's what I thought :-)

    About the BER, didn't know that. BER is the reason in would take raid 6 instead of 5, but that doesn't eliminate it ofc.
    a c 127 G Storage
    April 21, 2010 2:00:11 PM

    Indeed you add one parity disk per 4 disks; so the overhead would be 25%. However, please note that expanding this way is SAFE. Expanding a hardware RAID with one or more disks is incredibly risky.

    Because of disks getting bigger, there is an increased chance for additional bit errors on other disks when rebuilding, or even on all the disks. Not even RAID7 would protect you against that. RAID generally doesn't protect against corruption as well.

    So, if you want hardware RAID and use expansion, you also need a 1:1 copy of that data as backup; as expanding can cause all data on all disks to be destroyed. If you don't believe this you may be in for a surprise one day.

    The expansion procedure would relocate all data; each byte will be moved to other disks. This will only work right if all disks are 100% free of any errors. If any severe error pops up, it is likely you lost all data during the expansion.

    About your last example, in my maths (5-2) * 2 = 6TB; with another 5 added you have (10-2) * 2 = 16TB. Now with ZFS that would be 5-1=8TB and 10-2 = 16TB. Only if you use RAID-Z2 will you lose more disks; but of course you will use RAID-Z when adding disks per 4.
    April 21, 2010 4:12:36 PM

    Thanks for the reply again :-) all very interesting.

    I would however use software raid (using linux's mdadm) , not hardware raid. I don't see any reason why I would use hardware raid.

    I realize that ZFS with it's self healing etc really has something other file systems can only dream about, but still, it doesn't have the means to expand like mdadm can.

    Ideally I think I can get to 15 data disks in 3 steps, 5 initial (6 TB in raid 6), 9-10 after 6-12 months maybe (16 TB in raid 6, or maybe 14 if I use one as hotswap), and 15 after 18-24 months more or less (26 TB or 24 with the one hotswap). If I wanted to do the same with ZFS (using raidZ2) I would end up with 18 TB total (not using hot swap drives), which isn't bad, but it's at least 6 TB under what mdadm can do.

    Maybe those few extra disks are worth it, considering what ZFS can do for you. Maybe it would be better if I started off with 8 disks, and then add another 8 later on, at least that would make it more comparable to the array size mdadm does, but it would mean buying 3 additional hard disks, and a pci-e controller to hook them up. Which would put me close to 1800-1900 euro initial I think, which is a lot of money. Maybe I'll drop the expensive hotswap bays and spend that money on this.

    I'll think about it :-)

    I'll try getting it to work with VMware first ofc, never installed freeBSD before, so don't know where that will take me.
    April 21, 2010 7:52:50 PM

    Changed my hw setup a little to:
    CPU: AMD Athlon II X2 240e 70 eur
    Mobo: ASUS M4A89GTD PRO 100 eur
    Memory: 2x Kingston 2GB 1066 Mhz DDR3 ECC 2x70 eur
    OS HD: Western Digital Scorpio Black WD1600BEKT 50 eur
    SATA Addon: Supermicro AOC-SASLP-MV8 120 eur
    Raid: 8x WD Caviar GreenPower WD20EARS, 2TB 8x 125 eur
    PSU: 100 eur ?
    Case: LIAN-LI PC-A77F 240 eur
    HD Bay: 4x Coolermaster 4-in-3 4x 20 eur

    Which totals to 1900 eur (tax incl), about 160 euro for each tb (12 initial), I dont know weather that ratio is either good or bad, but it sure aint cheap :) 

    Open to cheaper suggestions, especially the chassis is expensive imo, 240 euro to get 12 5.25 bays and then another 80 to convert them to 3.5 bays.

    Just for the record, the "2nd phase" to add another 8 disks would cost another 1120 eur, so little over 3k total vor 24 tb of redundant space
    a c 127 G Storage
    April 21, 2010 10:45:09 PM

    If you like some FreeBSD+ZFS guides, i just written these:
    http://submesa.com/data/bsd

    Complete with screenshots and step-by-step instructions. :) 
    April 23, 2010 6:25:54 PM

    sub mesa said:
    If you like some FreeBSD+ZFS guides, i just written these:
    http://submesa.com/data/bsd

    Complete with screenshots and step-by-step instructions. :) 


    Great guides, srsly
    Im setting it up on vmware at the moment, all goes well :) 

    I suppose FreeBSD can run most of the linux software aswell? I was planning to use the NAS as nas (naturally) with smb shared etc, torrent client with transmission daemon (or similar), webmin for easy management, itunes server (firefly) and maybe some other stuff later like DLNA, ...

    Any suggestions to what software should be used on bsd? Im familiar with all of the above (except dlna) on linux
    a c 127 G Storage
    April 23, 2010 6:46:35 PM

    FreeBSD is Linux binary compatible, meaning it can run most compiled Linux binaries. However, most linux-software is open source and thus the source code is available. Those are typically in the ports-tree; which my setup guide will install.

    So for the transmission-daemon, you would use:
    /usr/ports/net-p2p/transmission-daemon
    make install clean

    and perhaps also:
    /usr/ports/www/transmission-web
    make install clean

    if you want web-interface. I have no experience with iTunes or DLNA, though.

    I think webmin requires you to setup the system in a much different fashion; you may have to change a lot to get this working. I suggest trying to learn the command line interface; once things are running it should be maintenance-free.

    If you really want a good web-interface i would look at FreeNAS 0.8 for example, not yet released but would provide ZFS v13 just like FreeBSD 8.0. The older FreeNAS 0.7.x is based on FreeBSD 7.2 which has ZFS v6.
    April 23, 2010 8:39:10 PM

    Looks nice, I'll try getting the transmission on next :)  that basically is the main thing. The iTunes and other stuff is just nice to have :) 

    Ive experimented with FreeNAS for a while, seemed to work fine, few drawbacks but almot 0 configuration. I think if it was on ZFS 13 it would be a no brainer :) 

    I did manage to get Webmin installed, but with very limited functionality, so there really isn't a point in having it at all actually...

    One problem tho, but its probably related to Parallels (the virtualisation software I use, like VMWare but for Mac). You basically have the choice to connect either virtual IDE devices, or SCSI devices. I added the OS HD as IDE device, which is working fine, but when I add 1 or more SCSI devices, to simulate the raid array, at startup I keep getting some error message:

    bt0: bt_cmd - Discarded reply data byte for opcode 0xd

    I suppose it has something to do with a missing driver orso. I've tried disabling the AHCI thing (in your config guide), but that didn't change anything. For the record I couldn't get Parallels to work with Freenas either, so I guess Ill boot up my Windows with VMware and see what that tells me.

    About FreeNAS, do you think the .8 version would be released shortly? I think they just released their .7.1, I'm not sure but I don't think the .8 will be released tomorrow :) 

    [Edit] Just read some more info about this on their forums, and there are nightly builds (testing) available for a .7.2 version, which is based on FreeBSD 7.3, which would also include the ZFS at version 13. Probably not suited for a live environment but it may be available sooner then I thought

    Thanks again for all the help
    a c 127 G Storage
    April 23, 2010 9:50:49 PM

    The new AHCI driver in my config guide is only active when working on AHCI controllers. If the controller is legacy IDE mode, the old ata driver is used - which does not support NCQ or hot-plug.

    If VMWare does not allow AHCI controller, you may want to try Virtualbox instead. It is free (both open source and commercial packages) and is quite fast. You can download the Mac OSX image here:
    http://download.virtualbox.org/virtualbox/3.1.6/Virtual...

    Also, Virtualbox allows to add physical disks, in order to reduce the overhead which is greatly desired when running things like ZFS. You can create a raw disk using this command:

    VBoxManage internalcommands createrawvmdk -filename <Absolute Path to output File> -rawdisk /dev/sda -register

    Change the path to output file and disk name (/dev/sda) to appropriate choice; this is done on the host system. Now add the .vmdk in Virtualbox and attach to your VM with AHCI SATA controller. Then use the ahci_load="YES" in /boot/loader.conf to use the new AHCI driver in FreeBSD.
    May 1, 2010 9:43:11 PM

    Ordered my hardware today :) 

    One question tho:
    I was thinking to start off with 6 drives in RaidZ, then afterwards (when more space is required), add 8 more drives. Use 7 of these new drives to create a new RaidZ2 array.

    Once this is done, move all the data from the original array to the second one, delete the original one, recreate the original one as RaidZ2 (with the 1 spare drive) and then "join" the 2 arrays together.

    PS: I tried doing this in freenas, and it did work when doing it in command line. the GUI didnt allow me to create a second pool.

    Would this be possible? And if yes, would you recommend doing this, or are there reasons not to do this, or better alternatives?
    a c 127 G Storage
    May 2, 2010 5:26:05 PM

    Yes that could work. Essentially you're creating a different array, copying data over, then expand the pool by adding another 7-disk array (the original 6-disk array).

    Just make sure you don't make any mistakes and do a scrub on both arrays to ensure all checksums are correct and thus your data is transferred correctly.
    May 2, 2010 8:00:19 PM

    Thanks, and will do the scrub :) 
    March 15, 2011 6:01:04 PM

    Hi Sub Mesa You have some really good advice here. Thank You.
    I'm looking for just this kind of low-power solution for a ZFS box. But I'm having a bit of a problem finding the right AMD CPU and MOBO. We're now in '11 so do you happen to know the right components to get?

    Thanks again.

    PS - only difference is I'm putting in 8 hot swap drives (Sammy F4's, low power).

    sub mesa said:
    Low-power means something like 35-50W excluding disks. This is easily achieveable with an AMD setup and efficient power supply.

    For example:
  • AMD Dualcore with 35W or 45W tdp means ~2-3W idle
  • AMD 740G/780G/785G chipset means ~2W idle
  • Amacrox 400W fanless PSU is 90%+ efficient, one of the most power efficient power supplies and comes without a fan

    If you go for 2,5" HDDs you can even go for an "extreme" setup with PicoPSU power supply, which is like ~97% efficient and can reduce idle power consumption to 30-35W, while still providing high performance with lots of memory and dualcore chip to saturate your gigabit NIC.

    Note that TDP can be misleading; while the maximum power consumption is important for chosing the correct power supply, its not important for actual power consumption, as your hardware will idle most of the time. In fact, all PCs in the world idle more than 90% of the time. Idle power consumption is what you should look at if you are concerned with the environment and/or want to save on energy costs.

    May 11, 2011 6:40:04 AM

    I have an old Server Mainboard with 16 sata ports and 7x PCI Slots and wanting to make some use of it

    I am Thinking of getting 7x old PCI 8 port Sata Cards and throwing in all my old HDD's 500 ~ 1tb size range

    What I am wanting is something like this

    Expandable Volume

    A way to Expand Array with Redundancy with out Rebuilding the Entire Array
    ie Old Data Stays where it is and when Extra Drives are Added the Redundancy lies within their Group but the Volume Expands ( like how M$ has the Option to Expand a Volume with logical Drives )

    That way if their is a Failure only Data in the Group has a possibility of being Corrupted and the Rest are still Fine


    I am thinking maybe a mixture of JBOD with Raid 5

    ie

    Group 1 8x Raid 5 Mode HDDs
    Group 2 8x Raid 5 Mode HDDs
    Group 3 8x Raid 5 Mode HDDs
    Group 4 8x Raid 5 Mode HDDs

    Volume 1 Consists of JBOD Groups 1 ~ 4

    as new Groups are Added new Data is written to Blank Space


    OS with WOL ( so it St3's when not in use and wakes up on LAN packet )

    But here is the Big one ... Power management that Spins Down the HDD's
    and Alows Folder Browsing with out Turning on every damn HDD on system
    would be great if only the HDD's Containing the data where the only ones that powered up


    to SUM it up needs to cut back on Power Bill and Running up to 70 odd HDD's 24 / 7 aint going to help

    also wanting to leave Hardware Raid .. the Cards just Die and are a pain to get replacements for when you need them !!!
    !