Motherboard integrated RAID5 performance

[PetyaMaster]

Hello everybody,

For a long time, I was wondering about a safe and also fast storage solution that's protected against drive failures. The solution of choice would be RAID5.
Since I'm not a company but an individual, high-priced server solutions were out of scope. In the last months, several motherboards came out with RAID5 capable controller chips integrated on them. As my current everyday used config is almost 3 years old (Socket 462), it is an attractive alternative for me to change the whole platform also.
I know that RAID5 needs parity calculation on the fly, and in case of motherboard integrated controllers, this task is done by the main processor, as it's also the case with cheap standalone controller cards. This solution is less-performing and more resource-consuming by its nature. But the question is : how much less performing ? I want the RAID5 solution having the best price/performance ratio.
I've searched the net a lot for integrated mobo RAID5 reviews, found none.
I'm especially interested in the performance of the following chipset solutions :
- nVidia 430 SouthBridge
- ULI 1575 SB, 1597 NB
- ATI's still-not-released new SB chip SB600
- Intel Matrix Storage (ICH6R/ICH7R)
If anyone has info or can link to sites having RAID5 performance comparisons regarding these chipsets, please post that here.

Cheers,
Peter

 

[enewmen]

I think it's going to be a while before hardware accelerated RAID 5 becomes common on motherboards.
I have no idea when the nForce5 or the Intel 1000 series chipsets come out. I didn't even hear rumours on this yet.

 

[PetyaMaster]

wusy :
First, thanks for the useful infos.
Second, I've finally found two reviews :
http://www.techreport.com/reviews/2005q4/chipset-raid/index.x?pg=1
http://www.gamepc.com/labs/view_content.asp?id=raid505&page=1
but neither are comprehensive (like Toms' )
Looking at these and summing up all the infos I have I got some questions left :
1. Out of the first review, it seems neither chipset integrated RAID5 solutions use the system processor for parity calculation, but the chipset itself does it. 'cos CPU usage is about 2-4%. Is this really true ? It's hard to believe ...
2. Is writing so incredibly slow with addon hardware RAID5 cards compared to reading also ?
3. With sofware RAID5 (in case of chipsets), write performance equals to processor speed, and CPU is always 100% used during disk writes, isn't it ?
4. I suppose while reading in RAID5, parity data isn't used to optimize drive head movement, is it ? I mean that the controller reads the parity chunk instead of one of the data chunks of a row of data and calculates the missing data out of parity in order to minimize head movement. They don't do it, do they ?
Thanks for the answers in advance.

Cheers,
Peter

 

[AdrianX]

Hi, I have an ASUS A8N-SLI DELUX with an 3800x2 CPU. I use the onboard SATA RAID5 controller (Silicon Image 3114) for RAID5 with 4 HD's. I'm using 4x160 Maxtor SATA drives.

http://www.adrianjudd.com/~adrianx/photos/raid/RAID 5 Silicon Image 4 drives.jpg

The above gives the results I get. It's fine for read, but terrible for writes. Also the random reads are way down too. Now there are 2 reasons for this, a) there is no hardware XOR and b) there is no hardware caching.

I now have an LSI PCI-X RAID5 controller with hardware XOR and 128MB cache. I'll run some results when it's all together. But I am expecting 4-8x write improvement.

OK, back to RAID5 on north/southbridges. Well, I'm looking forward to see what happens, but if you want decent performance then don't hold your breath. For a start they aren't going to have any decent onboard cache and I can't expect any decent cache algorithms either. Secondly, they aren't ever going to be able to match an Intel XScale CPU witch drive the latest RAID5 controllers from Promise, Areca for XOR etc.

So, where does this leave us? Well, yes we could get some half decent RAID5 with the next NForce chipset. But, it's not going to be in the same league as a dedicated hardware solution, because the dedicated hardware solution is more complex than a north/south bridge.

Now, if NVidia slap a XScale CPU onto their chipsets a la dual core and allow some cache on the m/b, then that would be great. But, I can't see that happening.

 

[TabrisDarkPeace]

nVidia run 'nvraidservice.exe' for 'RAID-5' on nForce 4 boards.

If you have a decent dual/quad-core system, and using 'onboard RAID-5' then the XOR calculations are just passed to idle CPUs. Unless it is like an Adaptec RAID-5 onboard with a dedicated XOR processor and RAM.

With enough RAM for the OS (Write-Back) Disk Cache write performance won't be that slow, as 1 GB of writes could be write-back cached by the OS. Then a say 2 GHz core can work its 'XOR magic'. It'll still be a hell'uv'alot faster than a single drive.

RAID-5 is best with 4 or more HDDs, but starts to excel at 6 HDDs. It'll work with 3 HDDs though, just performance with 3 isn't that great.
At 8+ HDDs consider moving to RAID-6 in case two HDDs fail at once.

On my system, which has being online for 12.75 CPU hours (3 hours 12 min) currently, the 'nvraidservice.exe' has registered 0:00:00 CPU usage. As in 'under 1 sec of CPU time' over 3+ hours of use.

For reference:

- Valves 'Steam.exe' has registered 7 sec CPU time and done nothing really in comparison. (Just sitting idle, checking for updates, authenticating, and loading their basic GUI into memory).

- Firefox has registered 0:02:00 just now, 2 min of CPU time.

As you can see 'nvraidservice.exe' is not 'anywhere near as bad, esp for the price, as people claim it is.

Do remember that over 80% of HDD I/O is reads, and not writes, so XOR overheads do not come into play (with just reads). Loading games is 'reads' btw.

If you are looking to save cash and want RAID-5 features, then yes, it is a very smart thing to do having RAID XOR performed by the processor(s) core(s) that spend 95% of their time idle anyway.

 

[AdrianX]

nVidia run 'nvraidservice.exe' for 'RAID-5' on nForce 4 boards.

If you have a decent dual/quad-core system, and using 'onboard RAID-5' then the XOR calculations are just passed to idle CPUs. Unless it is like an Adaptec RAID-5 onboard with a dedicated XOR processor and RAM.

With enough RAM for the OS (Write-Back) Disk Cache write performance won't be that slow, as 1 GB of writes could be write-back cached by the OS. Then a say 2 GHz core can work its 'XOR magic'. It'll still be a hell'uv'alot faster than a single drive.

RAID-5 is best with 4 or more HDDs, but starts to excel at 6 HDDs. It'll work with 3 HDDs though, just performance with 3 isn't that great.
At 8+ HDDs consider moving to RAID-6 in case two HDDs fail at once.

On my system, which has being online for 12.75 CPU hours (3 hours 12 min) currently, the 'nvraidservice.exe' has registered 0:00:00 CPU usage. As in 'under 1 sec of CPU time' over 3+ hours of use.

For reference:

- Valves 'Steam.exe' has registered 7 sec CPU time and done nothing really in comparison. (Just sitting idle, checking for updates, authenticating, and loading their basic GUI into memory).

- Firefox has registered 0:02:00 just now, 2 min of CPU time.

As you can see 'nvraidservice.exe' is not 'anywhere near as bad, esp for the price, as people claim it is.

Do remember that over 80% of HDD I/O is reads, and not writes, so XOR overheads do not come into play (with just reads). Loading games is 'reads' btw.

If you are looking to save cash and want RAID-5 features, then yes, it is a very smart thing to do having RAID XOR performed by the processor(s) core(s) that spend 95% of their time idle anyway.

what performance do you get from the NVidia software RAID5? with 4 HD's i'd ideally like read to be the same as RAID0 with 3 HD's and write to be like a single HD. With the Silicon Image chip and software is's no where near this.

i wasn't aware there was a RAID5 driver for my m/b. but it might be worth trying it. unfortunately, i can't use NVidia RAID0 with my SCSI card - stops the SCSI card from booting *sigh*

 

[TabrisDarkPeace]

Wtf ?, Just change the boot device order

No board I am aware of has limitations like 'order of 3rd party bootable devices', not even some rather old ones.

 

[PetyaMaster]

AdrianX :
Your scores with the Sil3114 are really too low even for reading. This could be because ASUS may have connected the chip to the PCI bus. Check out that is this the case or not. Probably is :-(

Thanks everybody for contribution.
One thing still isn't clear for me. In case of chipset intergated scenarios, what calculates the parity ?
A, the chipset itself
B, the main processor
If A, then is it possible to move the calculations from the chipset onto the processor via a driver or not ?
If B, then why just 20-40% is used for 10-20 MB/s data transfer ? Isn't it configurable to use more CPU time for faster transfer ?

Cheers,
Peter

 

[TabrisDarkPeace]

Isn't it smarter just to use ALL AVAILABLE PROCESSING POWER, when required for XOR data generation (it ain't parity), during write ops only ?

First law of computing: An application / process will expand to use all available processing power unless something holds it back. In this case the 4 x HDDs can only sustain 160 MB/sec or so of writes, 25% of which is XOR data, spread across ALL drives to improve write performance. Generating 40 MB/sec of XOR data for todays processors is nothing overly taxing to be honest.

A SHA512 / MD5 algorithm will burn more CPU time just performing reads from the HDD and processing it into a hash. XOR is a pretty basic function it would only take like 1-2 clock cycles (per X bytes) in todays chips surely.

RAID-3 is the one with parity that writes like a dog, you might have been given bad advice somewhere :?:

That way if the CPU is busy it'll write slower... maybe... but when the CPU isn't busy it'll fly. Bearing in mind with a dual-core one core is almost always 50% idle anyway.

You can set the priority of 'nvraidservice.exe' lower to get more CPU time dedicated to foreground apps, or to realtime for maximum write speed performance at all times. Frankly it makes jack all difference as it uses so little CPU time anyway, esp with a dual-core processor, or multiple processors. 8)

The main thing to remember is to get 2 GB+ RAM, as OS Disk Caches these days are reasonably advanced, and will perform write-back delayed, write combining caching.... although Norton Cache / Microsoft SmartDrive did this aswell.... 20 or so years ago. , More RAM = More memory available at a given momment for write back caching if doing heavy writes.

 

[AdrianX]

TabrisDarkPeace: "Wtf ?, Just change the boot device order"

lol, well it's SCSI boot drive. Sil3114 RAID5 works, NVidia RAID locks the pc after post. Just an incompatibility. Seeing as I didn't know I could go with RAID5 on NForce4, then I've not tried to fix it. I might now have a second attempt.

PetyaMaster : "Your scores with the Sil3114 are really too low even for reading"

Yes, it's Sil3114 off the PCI , unforunately. So, I wouldn't expect more than 100MB/sec. Still that's ok as it's on par with RAID0 with 2 drives, maybe a little faster. It's the slowness of writes that's the issue. I can't see/do anything to speed it up :-(


Now, seeing as my pc is for image editing, write speed is way important. read, edit, save, edit, save, edit, save ... So I have more writes than reads. My plan was to save up for a Promise PCIe SATA controller and then up the drives to rapters. But, this is a long way off as it's damn expensive. The 4x 160Gig HD's are <£200, so using the Sil3114 controller was a cheap option. Alas, it's sh1t.

 

[TabrisDarkPeace]

Are the files you are working on over 80 MB ?, (once saved and LZ77 compressed, etc)

Most 'work in progress / high res edit' formats use LZ77 compression, and it taxes the CPU and 'slows' writes down when performing saves (as has to process LZ77 each time).

My image save time (for 4096 x 4096 images, multiple layers, etc) is long but not because of the HDD, it is because I use LZ77 to pack the file sizes down and it hits CPUs. If I turn LZ77 off the saves actually speed up.

Hell some of my images are 16 MB files, but they 'save' at about 1 MB/sec, indicating the bottleneck ain't my HDDs. (4 x Maxtor Diamond Max 300 GB, 16 MB cache each, in RAID-0 can save faster than that 8) ).

PS: The reason loading 2 RAID BIOS will lock the PC is that they are trying to 'share' resources or map the same memory. This is far nicer than corrupting everything on the array(s). , You should be able to reconfigure at least one of them to not try and do this.

 

[AdrianX]

Yes files are over 80Meg, but I usually don't save with any compression. RAID0 is fast, RAID10 is fine too. Just RAID5 that is so slow

As I said, I hadn't realised I could get NForce4 RAID5 - Asus drivers with the m/b only supported 0 and 1. BIOS only supported 0 and 1 too. Now, I can get NForce SATA to work, just not RAID. So, if the software RAID5 uses the non RAID BIOS option of the m/b then I'm fine.

I'd like to move away from the Sil3114 controller as being PCI isn't going to share bandwidth with my SCSI controller. Currently, I have a single 15K SCSI HD, but I plan to add another for RAID0. This will then excede PCI bandwidth and present a major bottleneck with the Sil3114...

So you have any benchmarks with NForce4 RAID5? I use SSandra as the sequential read/write seem to relate well to Photoshop saves, without compression.

 

[Gintux]

The answer is found by the self - how the RAID5 actualy works. If system writes even a bit to volume, this means, that controler must:
1) write data to involved in data change disks,
2) read a stripes (generaly stripe size is 64K) from all disks - 1 (parity involved) in volume,
3) do a XOR parity calculation,
4)write calculated parity.
If XOR is done by CPU this means all this burden must pass over all the system (SATA/IDE bus, chipsets) to RAM involving CPU for XOR. For intensive data processing systems (database, video processing-rendering, etc. servers) this is unacceptable. There your need inteligent RAID adapter with dedicated XOR processor and onboard cache (the more your have the more you gain ). For database servers backup battery is a requirement. Some RAID controlers does not allow you to activete "Full chache" without them.
For slow data changing systems (file servers, video storages etc.) scenario described above should be not much of burden.