Raid-Z speeds are slower than single SATA drive

[freebagel]

I recently set up a Raid-z array in FreeNAS 0.7, that will connect to my computers through a gigabit network. I'm using 5x2tb western digital eads drives.

I ran atto on the array to test read/write speeds, and they were quite a bit lower than I expected (about 30-40mb/s write, 40-70mb/s read).

To make sure it wasn't a network issue, I ran the same test on a single SATA drive in the same file server that wasn't set up as part of the array, and got much better speeds.

Now, I wasn't expecting the world with Raid-Z in terms of speed, but it's my understanding that performance should at least be better than a single drive, and should come fairly close to maxing out my gigabit network.

The hardware I'm using for the server is a core i3 processor with 4gb ram, so that shouldn't be bottlenecking things at all.

So what's the slowdown here?

 

[freebagel]

Even more baffling is that even if I change this to striping (basically Raid-0), my read/write speeds are slower than the single drive.

 

[adampower]

Do you have your 5 drives on a 'Hardware RAID' pci card? Can you please give us some hardware and setup information?

 

[505090]

my first thought is you are probably running some older hardware and using the MB's sata ports instead of a raid card. so the performance drop you are experiencing is most likely that due to all the raid calculations required for each read and write to the array.

 

[adampower]

I think the core i3 + 4gb ddr3 should push the parity AND error correcting using freenas. Although, are you using embedded? or "full". I've heard problems running freenas from an HDD.

The other thing is that Freenas .7 uses ZFS ver. 6 whereas FreeBSD 8.0 and above use ver 13 or 14.

sub mesa has some freebsd tutorials on his site. Maybe check into scrapping freenas and using FreeBSD raw with his tutorials. Check any of sub mesa's replies and there's a link to his site from his avatar.

Freenas is supposed to be coming out with 0.8 soon which will be based on FreeBSD 8+ and should incorporate zfs ver 14.

Also, Win7 Ultimate has the ability to use NFS file systems and this may help. Samba often has speed issues. This is troubling though.

I'm following this thread closely as I would like to use this same setup in the near future (my thread is something about a system that will max gigabit lan)

 

[505090]

I think the core i3 + 4gb ddr3 should push the parity AND error correcting using freenas.

and you got those specs from where...

 

[adampower]

The hardware I'm using for the server is a core i3 processor with 4gb ram, so that shouldn't be bottlenecking things at all.

I don't know how to use the quote system properly. But, this is from near the bottom of the original post. It seems I assumed ddr3 memory but I don't think there are any 1156 boards that use anything less.

 

[505090]

to quote someone just click the "reply to xxxxxx" in the bottom right corner of each post

I see it now, guess i need to read a little more thoroughly

 

[freebagel]

Do you have your 5 drives on a 'Hardware RAID' pci card? Can you please give us some hardware and setup information?

The motherboard is a gigabyte GA-H57M-USB3.
NIC is Realtek 8111D I believe
Intel core i3 processor
4gb ddr3 ram
Client OS is windows 7
Server is running FreeNAS installed from FreeNAS-i386-LiveCD-0.7.1.5127.iso (the latest release)

The drives are plugged into the SATA ports of the motherboard, no separate raid card. I realize the hardware raid is preferable, but even in software I don't think my speeds should be anywhere near as low as they are now (especially in raid-0).

Previously I had this same setup running but instead of FreeNAS I was running Intel ICH10r raid, and was pulling speeds around 70-80mb/s in raid-5 and speeds that more than maxed out my gigabit connection in raid-0. FreeNAS is supposed to be even faster than ich10r, yet now when using FreeNAS with the same setup I'm getting speeds less than half as fast in raid-z or raid-5 and speeds about 20% as fast in raid-0, again all below even the speed of a lone SATA hard drive.

 

[adampower]

Ahhh very good. Nice setup! So you installed freenas from the livecd. Do you have it 'embedded' on a usb stick? Or, it is possible to put an embedded version on a HD.

It seems to me the ICH8,9,10 soft raid systems compare very favourably to so-called hardware raid. But I agree that FreeNAS should outperform on a system such as yours. Have you thought of trying Geom raid 5? The problem might be with the older version of ZFS. Or maybe you would have to purge some important data to start again?

Ever thought of using FreeBSD 8+ and setting it up yourself?

 

[freebagel]

Yes, it is possible to install the 'embedded' version on an HDD. However I think I selected "full" install when I installed the OS, though I can't remember for sure. What's the difference?

I'm still in the testing phases so I don't have any important data on here yet, and can tear it down and try different things as much as I need to. By Geom raid 5 do you just mean FreeNAS's built-in software raid 5? If so, I have tried that, as well as raid 0, and am still getting speeds below a single drive.

As far as setting FreeBSD 8+ up myself, I think I would be in over my head doing that.

 

[adampower]

Ahhh yes... On one of your menus there will be the option to 'Upgrade to Embedded'. This option may come when you boot from the live cd. I recommend you put it on a usb stick or something similar as the raid will be affected by the partitioned hard drive that has the OS on it.

The difference is that FreeNAS is designed to 'embed' itself in ram. The OS is something like 64MB and when you turn the computer on, or off, it saves any changes to the disk. Other than that it is able to do everything in RAM. I think there is some issue with the fact that it needs to read and right info to the OS while operating. If this OS is on a hard disk it slows things down. Embedded is truly an upgrade. The term 'FULL version' is silly and they shouldn't use it.

Even if you upgrade and boot the embedded version from an HD you will immediately see a huge performance gain.

I know what you mean with the FreeBSD setup. Even with sub's site I'm a little leary. Freenas will upgrade soon and I think that's good enough for me. If my wife gives me a weekend to play with it I might try the FreeBSD ZFS config but it may be in 2011 or 2012.

Good luck. Please, upgrade to embedded.

 

[sub mesa]

Can you test local performance by logging in with SSH and perfoming:

dd if=/dev/zero of=/path/to/ZFS/mountpoint/zerofile.000 bs=1m count=2000
(write speed)

dd if=/path/to/ZFS/mountpoint/zerofile.000 of=/dev/null bs=1m
(read speed)

Please do not make mistakes with the dd commands. ;-)

Also, i believe you are using Samba (SMB/CIFS) ? Could you test with NFS if your Windows 7 version supports that (it should be in add/remove windows components or something like that).

 

[sub mesa]

And are you using onboard ICH10R controller or are you using anything with PCI or PCI-X? In the latter case; switch to PCI-express. ZFS hates PCI.

 

[adampower]

And are you using onboard ICH10R controller or are you using anything with PCI or PCI-X? In the latter case; switch to PCI-express. ZFS hates PCI.

I was told everything is plugged into the onboard SATA ports.

 

[freebagel]

Yes, everything is plugged into the onboard SATA ports.

Sub, I'll give those suggestions a try when I get home tonight.

 

[freebagel]

Ok sub, when I ran those dd commands you gave me I got the following results.

Write: 217,244,078 bytes/sec (207 MB/s)
Read: 318,975,364 bytes/sec (304 MB/s)

So are those the read write speeds locally on the server? If so that would mean my 40-50 mb/s writes and 60-70 mb/s reads when going through the network are being throttled by the network? It seems odd that they would still be so much slower than a single drive copying through that same network.

I tried to set up NFS but could not get it working (enabled it in W7, set everything up in freenas, but couldn't get W7 to see the share). However those times I ran above with the dd command were using the share mounted as CIFS/SMB, is that taken into account in that check?

 

[sub mesa]

The dd command tests local performance; thus leaving the network layer out of it. It could be the network puts significant CPU load due to it being on PCI instead.

Is the network adapter onboard? In that case, a modern mobo would connect that to the chipset or PCI-express x1 lane.

It is kind of strange that a single disk would get you proper speeds; though.

Could you also test FTP performance? This should be quick to setup; and would allow you to test whether it is the SMB/CIFS protocol which is limiting your speeds or something else.

Could you also test using CrystalDiskMark/AS SSD on your mounted network volume?

 

[adampower]

Did you upgrade to Embedded?

 

[freebagel]

Did you upgrade to Embedded?

Yeah, I had previously upgraded to embedded but it was still embedded from the HD and it didn't really change anything. I went ahead and started all over, wiped the whole thing, and reinstalled embedded 64-bit version from a USB flash drive and my speeds have improved somewhat.

I'm now getting 70-80mb/s write and 80-90mb/s read, which is good. However with gigabit's limit being 125mb/s and my local read/write speeds being well over that max, is it too much to expect speeds nearer to that or are these the best speeds people are typically seeing?


Sub, crystal disk mark gave me about the same read/write speeds as atto. However, the odd thing is that now even with my improved speeds after reinstall it's still showing me speeds of 62r/47w, whereas atto and actual windows copy performance are giving me about 85r/70w.

I got the ftp set up no problem, but I have no idea how to use it from the command line or use it to measure anything. I guess I've got some reading to do on that front. Is there no other way to test the gigabit lan? I've seen some programs that do it but all the ones I've seen want you to install something on the server as well, which I'm assuming won't work in freenas.

 

[adampower]

Could be the Realtek 8111d or your switch or your nic in the desktop? I have an almost identical setup which I hope to finish soon and was expecting 100 or thereabouts. Was hoping for more of course.

Maybe there's some tweaking to be done to maximize memory use.

Also, I've heard of using a small SSD as cash for a zfs system.

 

[sub mesa]

I'm now getting 70-80mb/s write and 80-90mb/s read, which is good. However with gigabit's limit being 125mb/s and my local read/write speeds being well over that max, is it too much to expect speeds nearer to that or are these the best speeds people are typically seeing?

100MB/s is the practical maximum of gigabit; 125MB/s is RAW bandwidth; that includes TCP/IP overhead and Ethernet layer overhead.

I got the ftp set up no problem, but I have no idea how to use it from the command line or use it to measure anything. I guess I've got some reading to do on that front. Is there no other way to test the gigabit lan? I've seen some programs that do it but all the ones I've seen want you to install something on the server as well, which I'm assuming won't work in freenas.

If you've got FTP running, simply start your Windows FTP client (FileZilla is an open source FTP client) and connect to it, then double click a file to transfer it and watch the speeds. Try this with a very large file (at least 2GB).

 

[freebagel]

When I transfer a file to the server through ftp, my write speeds hover out around 85mb/s on a ~5gb file.

 

[sub mesa]

Sounds right; probably network limited.

You could try a different benchmark: HDTune Pro (evaluation). It has the "File" benchmark, where you can set a delay in seconds. Set the delay to 30 seconds and size 1000MB; now start the benchmark and it should give you more consistent scores. I'd like to see the benchmark if you can post it online.

 

[freebagel]

For whatever reason, I cannot get HD Tune Pro to show my networked drives. I have them mapped as network drives, but only the local drives are showing up in it.