RAID 0 - not getting the performance I'd expect

jacksilver12

Distinguished
May 16, 2011
7
0
18,510
Hello,

I'm trying to identify a problem in my RAID setup.

I've setup a software RAID 0 using 8 x 1 TB 7,200 Samsung F1 drives, 32MB cache.

I'm interesting in video editing and working with uncompressed streams of HD (Quicktime or Image sequences). These need roughly 250MB/s

When I measure the speed of my Raid using HDD testing software like Sandra, it comes in at about 260MB/s

My question is, how do I improve this speed? Add new drives?

What is the limitation? Is it my CPU, or my RAM speed?

Should I be using my hardware raid card (I have a 5 port hardware raid -- so could have a 5TB RAID 0)? Will I see any improvement with hardware, over software?

I get pretty good results with certain things (e.g. copying a file from a USB drive to the raid is 11 times faster when compared to copying to the boot drive -- 5 seconds versus 55 seconds for the same file from a USB stick).

However, I'm not getting the performance I would expect when attempting playback fo tiff/dpx sequences, or other uncompressed video files, which is surprising, so I've started to try to isolate where the bottleneck is.

I've posted questions elsewhere with specific questions about the RAM and the CPUs... so just wondering if any kind and smart readers of the RAID sections would have any answers for me, specific to the RAID setup?

System specs:

S5000XVN Dual Mobo
2 x quad core 1.6ghz processors: E5310 quad core (8 cores)
8TB Samsung F1 drives, in raid 0 config
320GB Samsung F1 boot drive
16GB Fully buffered ram, PC2-5300F DDR2-666
Nvidia Quadro FX 4800 (1.5GB)

Core speed: 1596.1 mhz
Bus speed: 265.9 mhz
Rated FSB: 1063.7 mhz

DRAM freq: 332.4
FSBRAM: 4:5

HDD benchmark test on a file copy test gives me:

Size: 95828114
Time: 739 ms
Transfer Rate: 123.666 MB/s

On a create file bench test:

Size: 50331648 bytes
Time: 377 ms
Transfer Rate: 127.321 MB/s

On a read file bench test:

Size: 95828114 bytes
Time: 149 ms
Transfer Rate: 613.348 MB/s


Any thoughts or suggestions would be most welcome.


THANKS

Jack
 
What raid-0 does is alternate writing sets of data(stripes) on each of the drives in your array.

On a sequential raid-0 benchmark, you might get some impressive results. That is becase the benchmark is reading sequentially, and can overlap the reading of sucsessive
stripes from each drive. On writing, the default hard drive properties will be to not signal completion of the write, until the write is securely on the drive. That is to protect you from an error if the pc should fail at an inopportune time. You can turn off that checking if you want, and your writing should go faster. If.. your application will permit.

100-140mb/s is a typical hard drive sequential sustained transfer rate. There are some drives that do better.

If you can stand the expense, use a good SSD instead, you should be able to be 3x faster.
 

jacksilver12

Distinguished
May 16, 2011
7
0
18,510
Thanks!

Are you saying that it doesn't matter how many drives I add, I will only ever achieve 120-150MB/s ?

Or is there some other limiting factor?

What additional speed do I get by adding each drive? (I now have 8 in a RAID 0)?

Is there any additional benefit to be had to switching to a hardware array? (what speed could I expect from 5 x 1TB in a hardware raid, for example)?

If there is some other limiting factor... what is it? CPU? RAM?

SSD could be an option, but surely I can get a better speed than 125MB/s out of 8 x 1 TB in Raid 0??

What would I expect to get (transfer wise) from 3 X SSD for example?

THANKS

 
1) What you get now, is what you will get, regardless of the number of drives using the same application. If an application does not have the right access pattern, it is not something you can change.
2) The limiting factor is the mechanical characteristics of the drives you are using. The very best might be 25% better.
3) Sequential read benchmarks which will get better.
4) Hardware raid will help. Particularly an expensive one with a discrete processor chip, lots of cache, and bettery backup.
The battery backup will allow writes to signal completion immediately and improve write performance.
If the cache is large enough, you might get some more rereads from the cache.
5) If your sata is 3gb, you could get 300mb/sec. Intel 320 for that.
If you have 6mb sata, you might get 500mb/s. Intel 510 for that.
Raid-0 for a SSD will not help. Larger SSD's are a better option. Larger SSD's have more internal nand chips which can be read in parallel. An internal raid-0 if you
will.
6) How much drive space do you need? There are some very fast pci-e based solutions that are very expensive for smaller capacity needs.
7) What cpu and motherboard do you have?
 

jacksilver12

Distinguished
May 16, 2011
7
0
18,510
THANKS!

1) OK -- so it would make no difference if I dropped it down to only 2 drives, for example?
2) What speed can I expect from these particular drives... what are there mechanical characteristics? (Samsung F1 7,200 with 32MB cache)
3) How can I turn this on in an application? Where would I look? Can I turn it on as a general rule in Windows (Vista 64)?
4) What performance could I expect from a hardware raid? Will drive numbers make any difference here? I have an AMCC PPC405CR (https://myapm.apm.com/MyAMCC/jsp/public/productDetail/product_detail.jsp?productID=PPC405CR)
5) How do I work out if my sata is 3gb? Or 6gb?
6) I think I need 2TB of space
7) S5000XVN Dual Mobo
2 x quad core 1.6ghz processors: E5310 quad core (8 cores)
8TB Samsung F1 drives, in raid 0 config
320GB Samsung F1 boot drive
16GB Fully buffered ram, PC2-5300F DDR2-666
Nvidia Quadro FX 4800 (1.5GB)

THANKS AGAIN

 
your user manual should state whether you have sataII (3gb/s) or sataIII (6gb/s) ports on your motherboard.

while performance might be increased by ssd drives, don't be shocked by the incredibly high prices. this begs the question "is the performance really worth it?"

raid-0 for ssds might not help with speed but you should be able to use them in an array this way for increased capacity (your 2tb requirement). perhaps someone who has this setup can verify.

pci-e ssd solutions seem to have the best performance and capacities. although currently prices are absolutely outrageous. newegg has a 1tb model for a whopping $4,400. sequential read of up to 1.4gbps or so the specs said.

the only non-pci-e 1tb ssd listed had a maximum of 260mbps sequential read. newegg didn't have any in stock but the price was $2,200

----

because of the high prices, a full ssd system is not often recommended unless you have huge budget and require the fastest possible performance.

if you don't want to go with full ssd/hdd systems how about a hybrid?

you can keep current working files on your ssd array and older backup files on your hdd array. this would require much less space for the ssd drives drastically reducing expense. once a project is finished you could then move it to the hdd array to make room for new work.

would definitely be more cost effective. from a few quick google shopping searches it looks like a pci-e 240gb solution is about $550 and a 512gb sII solution is about $450.
 

jacksilver12

Distinguished
May 16, 2011
7
0
18,510
It's sataII.

Any ideas on the below:

1) What speed can I expect from these particular drives... what are there mechanical characteristics? (Samsung F1 7,200 with 32MB cache)
2) Would it make no difference if I dropped it down to only 2 drives, for example?
3) Turn off that checking -- How can I turn this off in an application? Where would I look? Or can I turn it off as a general rule in Windows (Vista 64)?
4) Could I expect a lift in speed by using a hardware raid? How can I calculate this?


Thanks
 

dfusco

Distinguished
Oct 29, 2008
379
0
18,790
2 x quad core 1.6ghz processors: E5310 quad core (8 cores)

Uh, no, it has 4 cores. It can handle 8 threads in some circumstances but it is a quad (4) core processor.
 

dfusco

Distinguished
Oct 29, 2008
379
0
18,790
oops I should really have coffee before I reply. You do have 2 quad core processors so yes 8 cores. Quite a rig you have there :)
 

jacksilver12

Distinguished
May 16, 2011
7
0
18,510
Hahaha, thanks.

But I'm wondering if that is the bottleneck, as well.

The effective speed is 1.6ghz. Not the 8 x 1.6ghz I'd like!

Software Raid 0 is presumably quite CPU intensive. So maybe that is where it is slowing down?

I just found an article which breaks down the speed of the raid like this:

Slowest drive in the raid 0 array, multiplied by the number of drives in the raid array, gives you the throughput. This would answer my initial question which is that I could gain extra speed by adding in additional drives...

A stripe set of four drives with a data transfer speed of 20 MB/s can therefore reach speeds of 80 MB/s, because data is read or written to or from the four drives in parallel.

So, with 8 x 1TB Samsung F1s (175 MB/s sustained, according to the write ups -- http://reviews.cnet.com/internal-hard-drives/samsung-spinpoint-f1-desktop/4505-9998_7-32733943.html) I should be getting more than enough! Like 1,400 MB/s !

But, I'm not getting anywhere near that...

Where could the bottleneck be?

 
Raid0 of multiple HHDs does nothing to improve access time, and for that reason improves random 4 K read/writes very LITTLE. This is what is so very important to boot time and program load times. What Raid0 does do is improve the sequential read/writes which is important when working with large files such as thoes 10 meg jpeg photes, video foles (DVD dot vob typically 1 gig, and a Blu-ray file can be up to 40 gigs), large spreadsheets, and large cad/cam drawing files.
I do not think your CPU, or memory is a bottleneck. Raid0 of a HDD will not saturate a SATA II interface, but More than two drives in raid0 may saturate a sata II port.

If you want to improve on access time of mechanical HDD, you could do a short stroke (Google shortstroke). On a pair of WD 640 gig Blk drives, it cut the access time from approx 12.6 millseconds to approx 9.5 mSec. - This is still 40->90 % slower than a single mid level SSD!!! Not with short stroke you gain performance, but LOSE capacity.

Added, raid0 is not linear in that although you should get a 2x improvement in seq performance with 2 drives, 3x for three drives - it does not deliever, you get a seq boost, but less than the theoritical boost Not remember, but maybe 60% ballpark in real life.

Use to use Raid0 exclusively, but with SSDs, there is just no comparisions. Raid0 only makes sense when working with large file types that I had mention.
 
There is no bottleneck. It is your application.

Sequential raid-0 benchmarks are specifically constructed to read and write large consecutive blocks of data with read ahead and write behind capability.
That is, the benchmark will read data in anticipation ahead of the time the application needs it . That allows the maximum amount of overlap of reading across your 8 drives simultaneously.

Unfortunately, most applications are more simple minded in design. It is easier to code that way.
Read some data, then process it before you read the next block. There is no anticipatory reading.

There may be some hardware raid controllers that have the smarts to detect such a sequential operation, and do the anticipatory read into their own cache. I have no details on such controllers, so do your own research there. They will be expensive.

The fastest part of a hard drive is the outer rings. If you planned on using just the first(outermost) 15% of the drive for your data, then you would get the fastest possible data transfer rates.

Maximum sustained throughput for a hard drive is hard to find. You need to go to the hard drive web site, and look for a download of the drive specification sheet.
As a rule, the denser drives, and the largest drives will be the fastest.

10K, 15K, and enterprise drives are tuned to maximum random iops, not sequential throughput. They may actually be slower than the spinpoint 1tb drive you referenced.
 

jacksilver12

Distinguished
May 16, 2011
7
0
18,510
Thanks for this. Very interesting and helpful. Apologies if my questions are a little oblique, but I am only learning!

I have two different Sata II cards, so perhaps the interface isn't the bottleneck.

I am dealing with large files. 8MB to 18MB per image, 24 images per second. So this is not the problem.

I will look into short stroke and see what I can do. Thanks for the recommendation.

So, on an 8 x 1TB, I should be getting approximately 8 x 175 (sustained transfer rate) x 0.6 = 840 MB/s?

Yet Sandra gives me 260MB/s which I desperately would like to double or triple!

J

 

jacksilver12

Distinguished
May 16, 2011
7
0
18,510
Thank...

"The fastest part of a hard drive is the outer rings. If you planned on using just the first(outermost) 15% of the drive for your data, then you would get the fastest possible data transfer rates."

I have no problem only using the first 15% of the drive.


 
260MB/sec is about the limit of sata2. Double that with sata3.
sata3 cards that go in a X1 slot are about $35. That might help.

You might look into a true raid card with a good sized cache and battery backup.
18mb is not such a large size, it will easily fit into a discrete raid card cache.

The fastest I/O is no I/O.
Can you add enough ram to your system to keep more of your work in ram and do less I/O?
Apps like photoshop benefit mightily from lots of ram.
 
geofelt,
Would a 8 gig Ramdrive be benificial as jacksilver12 has 16 gigs of ram. He could preload these images into the ramdrive (about 400 Images @ 18 Mb/image). A Ramdrive puts even an SSD to shame on throughput. I installed an 8 gig Ramdive (also have 16 gigs of ram) and ran a benchmark on it, but at work so can not provide the stats.
 

Good question.

It all depends on the app, and how it works.
The OP has a workstation motherboard with 8 ram slots, capable of 32gb of ram.
>16gb needs better than windows-7 home premium which is limited to 16gb.
A ramdrive setup could very be a real winner for the right app.
 
Took awhile but found it, Here is a comparision for a SATA II SSD and a RAMDRIVE (MY system).
Using AS SSD.
.................... SSD .............................. RamDrive
Seq ………… 201.74 …… 133.74 ………………. 4714.45 ….. 4966.82
4K …………. 22.79 ……. 80.33 ………………. 439.67 …. 353.32
4k-64Thrd… 125.25 ….. 119.26 ……..……….. 872.92 …… 855.73
Access time ..0.115 …. 0.206 ……………….. (LOL) ……….. 0.011ms
Score ……….. 168 …...… 213 ……………………… 1784 ………. 1706
…………………….. 473 …………………………………………… 4614

** (LOL) = Too dagburn fast, could not measure.
Could not locate the stats for Raid0 2 x 640 gig WD blks semi short stroked.