40Tb Server performance issue

[ashrafi]

I have build a media Server with following specifications,
Gigabyte GA-H55-UD3h motherboard
Processor i5 680
Dell Perc 6i Raid Card
Chenbro Expander
20 * 2Tb Western Digital Caviar Green in Raid 6 configuration
I have set Intellipark to 300secs as discussed on some forums , After installing Server 2008 R2 I transferred 3.4Tb data on the raid setup and it took 3 days . (The maximum speed reached is 30Mbps).I have set the cache block size to 512mb . I believe the speed if extremely slow , I made sure the raid card was installed in the x16 pcie slot , I might be missing something , how can I increase the performance of this media server or is it bottle neck due to hardware itself.

 

[FireWire2]

I have build a media Server with following specifications,
Gigabyte GA-H55-UD3h motherboard
Processor i5 680
Dell Perc 6i Raid Card
Chenbro Expander
20 * 2Tb Western Digital Caviar Green in Raid 6 configuration
I have set Intellipark to 300secs as discussed on some forums , After installing Server 2008 R2 I transferred 3.4Tb data on the raid setup and it took 3 days . (The maximum speed reached is 30Mbps).I have set the cache block size to 512mb . I believe the speed if extremely slow , I made sure the raid card was installed in the x16 pcie slot , I might be missing something , how can I increase the performance of this media server or is it bottle neck due to hardware itself.

Wow! It seems too costly

You need to see where is the problem of transfer is:

I would connect direct to another system (Window or MAC) via CAT5e cable minimum, then transfer DVD.ISO or BD.ISO files, you need to use file in range of GB

If the transfer is good then you have cable or hub/switches problem, otherwise take a close look at the system you just built, particular in network adapter property to see anything is out of ordinary

Here is the 40TB system that I built:
http://www.mpcclub.com/forum/showthread.php?t=22898&page=4

It transfers over 80MB/sec and only consume 185W max.

 

[ashrafi]

Amazing work , Can you give me details of your build which raid and expander did you use , secondly did you change the Intelli park and TLER on these WD green drives?
I need to optimize my system so I have correct cache stride /allocation size.
I have found out that I would need some sort of block alignment , not sure about it but your example would help greatly if you took these steps .
Thanks!!

 

[sminlal]

Write speeds will always be slow because of the need to recompute and rewrite the parity information on the array for each block that's written. But read speeds should be much faster (unless the array is still initializing or is operating in degraded mode).

With 20 drives in the array you may end up operating in degraded mode more often than you expect.

 

[mavroxur]

I have to agree, that write speed seems a little bit slow. I just configured a Dell R710 with a MD3000 disk array a few months ago, and built it as a pair of RAID 5's with a couple hot spares, and my write speeds were in the 90's. The MD3000 was full with (15) 1tb 7200rpm SAS drives, and was running on a PERC6 also. I'd check your array status. Now, of course, my speeds were gauged by copying from one internal array to the external MD3000 array. If you're copying over a LAN, then your speeds might be limited by several things. Your NIC, your switchgear, etc..... Connected with a gigabit connection, you should still be seeing transfer speeds in the neighborhood of 60MB/sec on a properly configured network (give or take, but with good switchgear and good hardware it should be in the high 70's or better)

 

[FireWire2]

Thank you!

Here are what I did

Yes, I was indeed thinking about how to built my NAS about two months. My goals are:
- GREEN (low power consumption).
- Fast so I can stream BD.ISO, MKV, DVD to my TVIX Slim S1 in my bed room and DUNE Prime in my living room.
- Connection must be Gb so I can ripped and transfer my BD.ISO in min, instead of hours or days.
- Low cost, otherwise my other significant won't approved .

After all research I end up with:

1x SuperMicro ITX Mobo X7SPA-O
1x Chasis http://www.norcotek.com/
4x SPM394 http://www.datoptic.com/ec/esata-hardware-raid-controller-spm394.html
1x 1GB USB falsh
1x 1GB DDR2 RAM
20x WD Green 2.0TB WD20EARS
1x 430W PSU


Why X7SPA-O
I select this Mobo because, it has built in Atom CPU, TWO GbE, PCI E 4x slot and SIX (6) SATAII ports.
I do not need dual, quad core CPU since the CPU only run network services at home, and these services would not be CPU intense
so fast CPU sort of waste energy and money. Matter of fact if there is a Celeron CPU in it I would use it
- Two Gb E - I can bond (LACP) it together to get double the bandwidth 2Gbps instead of Gb Ethernet, if needed
- Empty PCI E it allows me to add 8 more SATA ports if I ever need to expand the storage so the final port I will have is twelve (12) ports - 360TB RAW volume
Or add an additional NIC to increase the bandwidth of network
- Current six SATA ports allow me to connect up to 30x SATA Drives or 90TB raw (if i use the 3.0TB HDD)
- Built in CPU - since I will not have host CPU manage the RAID so DUO Core and other CPU is wasted
- Cost in high 100's

So for who wants to build a smaller system use ITX-220 - you can have a system go up to 30TB

RAID controller SPM394
I select this because it is a stand alone hardware raid. Once the raid configured, OS sees it as ONE BIG hard drive, NO drivers required, it means I dont have to worry about update or compatible issue
It rebuilt rate is over 200GB/hr - so with a volume of 15.0TB (5x3.0TB hdd) need to rebuilt it takes 7.5Hrs or 5hrs if 5x2.0TB is used
LCD display allows me to see what is going with the array instantly
Built in alarm - sound off if there something is going wrong
No problem with 2.TB - WD20EARS - I think WD30EZRSDTL would work just fine, but I did not have this drive in hand so - test it before use

WD20EARS
Low cost
Low power - when Write / read it consume ~ 6W
Seems to work fine with SPM394, the above raid controller - there is no problem found on TLER issue

Chasis http://www.norcotek.com/item_detail....delno=RPC-4220
Low cost - rack mount decent fans
20bay - rack mount

430W PSU
As I calculate the power consumption my NAS require about 200 max, therefore 430W PSU is more than enough
20x WD = 120W max + Mobo = 45W max + 4x SPM controller @ 15W Max + Misc fan/LED... = 200W max

OS: FreeNAS
Ultimate Low cost - $0.00
USB1.0GB flash cost $5.00

My NAS Front view

Building steps:

1_ Install controller
If there is FOUR 5.25 slots that would be perfect, but there is no such thing, I decide to cut the chassis's PCI slot sopen and drop four of these SPM394 into this corner of the chassis
with couple screws I was able to hold them in place

2_ Install MoBo+Drive+ and PSU
I drop the 1GB DDR2 in 1st then mount the Mobo the chassis, after that I install 20x HDD to the tray and inserted to the chassis, the PSU is going last, the sequence I have

3_ Prepare FreeNAS flash
I re-format the existing flash as FAT
Download freeNAS CD image here and burn it to CD
Boot from just burned CD
Install FreeNAS to USB flash - for more detail you can see it at freenas.org
Plug newly loaded FreeNAS USB flash to Mobo USB port (close to SATA ports)
Just google it or drop me an email

If you need more info I will update this post

4_ Connect raid controllers to Mobo
I only use four out of six Mobo SATA ports. There are TWO empty SATA ports, this allows me to expand another Ten HDD, if i want to - This is an immediate expansion option

After it's done, the system should have four SATA ports connected to four SPM394 and FOUR SPM394 SATA ports fan out to 20x WD HDD. An USB flash with freeNAS is connected to onboard USB port

(Continue...)

Edit: add more images

 

[FireWire2]

Part II

5_ Set up NAS

- Connect a keyboard and monitor - only for set up, after that system can run headless
- Power the system, go to BIOS change these:
- SATA II mode: Enhance or AHCI
- Boot drive: USB flash
- Disable Audio, Parallel port
Save setting and boot
Once it powered, configure four raid5 out of SPM394 via LCD panel, the end result Mobo will see 4x 8.0TB HDD, then we combine those to ONE big 32TB volume as RAID50

I'm configured this way, because the rebuilt time is minimal and one parity per five drive is lots more reliable then two parity for 20 drives (raid6).

I did a test on this RAID50 in my Win7 system, I get 840MB/sec, so the speed from the RAID is not a problem that I have to worry.

6_ Once it booted, the default IP address is 192.168.1.250. Using the connected KB change the IP to the existing network address

7_ Open Web Browser in your MAC/Window system, enter the new configured IP. Your should be able to log-in and configure the rest
With a little tune-up in SMB protocols, the system can easily transfer @ 80MB/sec or more

There are many features in FreeNAS, that I found useful for the home user:
Rsync - Remote backup MAC/Window data
iSCSI target - you can share the storage to your system
Bittorrent - Download your favorite media content


If you need help or have question, please post here, I will try my best to answer

FireWire

 

[ashrafi]

Nice post , In my honest opinion you could have saved lots of money if you used Expander card and connected External bays through it , I have not used this specific controller therefore I am unaware of features it offers , I was confused on how you individually configured Raid 5 (8 Tb ) initially and finally it was configured as 32Tb single volume , ( was it all controller based ? ..I mean i know how raid controllers have their bios and everything and it is easily configured ) .

Secondly , about inserting USB drive on the mobo , can you explain how you did that , ( i assume it always boots by default to the USB ).

LACP bonding , never experienced it myself , can you explain how your achieved it ?

"With a little tune-up in SMB protocols, the system can easily transfer @ 80MB/sec or more"..What tuneups did you do and how?

overall your setup is impressive , but while i read your post i wondered if this motherboard had the capacity to hold all this bandwidth ? I was trying to convince myself the upper $200 price for this motherboard would justify .
I ask this because earlier I had experience using Zotac Mini ITX for a small Storage-Server (10 TB ) , this specific motherboard had 6 sata ports as well , ( Processor price was def add up but I wanted to use decent i5 , further this mobo provided wifi , hdmi , Spdif)

 

[FireWire2]

Thank you

I'm not sure about expander that you're refer to. As far as i know, there are TWO type expanders: PM (Port Multiplier) and SAS expander.
These are just simple an one to many adapters, both does not offer raid function built in
- PM bridge is a 1 to 5 - there is NO hardware raid controller that supported PM, except SPM394
- SAS expander is a 1 to 16/24 ports - too expensive because to get it works you need SAS raid + SAS expander that would be over 1000's dollars

so for the cost i used SPM394 or SPM393. I have to configure 4x raid5 and combine it, because the SATA ports in Mother board are NOT Port multiplier supported, it SEES A SINGLE volume/drive only. All five drives configured as RAID5 in SPM394, it is treated as a SINGE drive by SATA host controller.

In other world I want five HDD through SPM394 shows up as a SINGLE drive in freeNAS, where raid5 function reliability and performance will handle by the SPM394 independently from FreeNAS.

There are pros and cons on my configuration
Pro: Low cost, easy maintenance, fast rebuilt. Most important there is no drivers or additional software needed, so if i update NAS firmware these raid arrays will not give me any compatible issue due to drivers like other raid card does

Con: loose 4 drives capacity in stead of 1 (8TB instead of 2TB), customize mounting these controllers

Note: Since SPM394 controls 5x HDD and presents to SATA host port as 8TB HDD, you need to make sure that SATA host port is not 2.0TB limited, don't be surprise if your board does, there are many SATA port that CAN NOT see a HDD over 2.0TB

Second...

After loaded freeNAS into USB flash, just connected it to USB port. Set up the BIOS to boot up from it - that is - this USB is an OS drive, like C drive in Windows
Once powered the OS (freeNAS) will be loaded run. there is another benefit that I have not mention, when runs freeNAS potential virus thread is minimized.
you can download this manual http://www.supermicro.com/manuals/motherboard/ICH9/MNL-1166.pdf. Search for "JUSB5", It will show you the connection

LACP...

please refer to: http://en.wikipedia.org/wiki/Link_Aggregation_Control_Protocol#Link_Aggregation_Control_Protocol

i wondered if this motherboard had the capacity to hold all this bandwidth ?

Yes it is, this board has an Intel I/O controller Hub ICH9R with GB of bandwidth internal and dual PCI express Intel NICs (one of the BEST NIC) built-in, I don't think you can get a better mobo for NAS than this board. Using Intel NIC alone, you would has 30% more transfer rate compare to Realtek NIC.

Of course you can use any the Mobo, but one use with FreeNAS, the HDMI, SPDIF, HiFi Audio, Wifi feature are not used. Matter of fact I disable everything in the BIOS: COM ports, Printer port, HD Audio...

Just keep in mind, the way I configure the raid array DOES NOT using Mother board CPU, the MAIN task for CPU is manage the network services, and this does not need a whole lot power . There is CPU graph in freeNAS that allows you to see CPU loading in real time. I have not able to load it to 100% on this Intel® Atom™ D510 processor .

Also when you using NAS

With a little tune-up in SMB protocols, the system can easily transfer @ 80MB/sec or more...

reading various post in internet - Here is my TUNE UP on our NAS:

1- Change Swap configured as: Type=File / Mount Point=Hardware Raid Volume / Size=512M
2- System | Advanced| sysctl.conf
- net.inet.tcp.recvspace = 524288
- net.inet.tcp.sendspace = 524288
- net.inet.tcp.delayed_ack=0

3 - Network | LAN Management
- MTU=9000
- Device polling=Checked

4 - Services|CIFS/SMB|Settings
- Large read/write = Checked
- Asynchronous I/O (AIO)=checked
- Auxiliary parameters:
deadtime = 1
getwd cache = yes
max xmit = 65535 # 32768 # 65535
read raw = yes
socket options = TCP_NODELAY
strict locking = no
write raw =yes

With these change I get over 80MB/sec

For whom who may not need 20bays system - look at this eBOX-N, this can handle 30TB (10x 3.0TB)

FireWire

 

[Seeking Info]

Superb Post, I am hoping you can clarify


FireWire2 wrote:

"Once it powered, configure four raid5 out of SPM394 via LCD panel, the end result Mobo will see 4x 8.0TB HDD, then we combine those to ONE big 32TB volume as RAID50"

ashrafi ask:

"I was confused on how you individually configured Raid 5 (8 Tb ) initially and finally it was configured as 32TBb single volume , ( was it all controller based ? ..I mean i know how raid controllers have their bios and everything and it is easily configured ) ."

I am guessing that you used the LCD control panel of each SPM394 Raid Controller to format, initialize and create a five disk Raid 5 Array on each SM394 controller. Is that correct ???

I am also guessing that you used the bios of your mother board to combine the four Raid 5 Arrays - with each array appearing as a single disk to the bios - into a "4 disk" Raid 0 array using the bios on the Mother Board to create the Raid 0 array, is this correct ???

I am now guessing the Raid 0 array now appears as a single 32TB hard disk to the operating system on the NAS computer you created, is this
correct ???

I am also guessing your reference to a Raid 50 array meant a Mother Board Bios Based raid 0 built on top of 4 independent SPM394 controller based Raid 5 arrays, is that correct ???

Again, great post.

 

[FireWire2]

Cost
If I use PCI Express it costs me at least $1450.00 for 20+ SATA port (1 year ago)
where my solution cost me merely $860.00 - near $600.00 saving where i can spend on HDD

Easy
With PCI express, I has to use CLI (command line interface), and not quite sure about loading the drivers in freeNAS

Where SPM394, which will group 5 drives as RAID5 (data protected) and present it as a SINGLE drive to the BIOS. Which is nice because there is no driver, software to install or messing around
Addition to:
a) MB SATA port does not support PM ware so there is no way I can see 20 drives in FreeNAS
b) Assume I can see 20x drive in and created a parity raid (raid5/6) - It will put a big dent to host CPU resource

So SPM394 is a best/simplest solution for my need

Here are the answer to your questions:

I am guessing that you ....

- Yes! with a couple button press I have 8TB volume under a minute

I am also guessing that ....

- No, i do not use the MB raid function because freeNAS not supported all MB raid, I just use it as AHCI, so FreeNAS will see four (4) 8TB drives.

In FreeNAS Disk Manger, then use JBOD* or RAID0 to combine it to 32TB volume.

Reliable

So far it runs 24/7 about 1 yr.


* this should be name as BIG or SPAN to be correct

 

[Cracktroz]

Hello! you need to format in 64k your raid and for more performance try iSCSI protocol with a gigabyte network i work with (HPC) in cluster and is the best way to work.

 

[Rusting In Peace]

1) I hope you need the 40TB capacity now because you should always minimise the number of disks in a raid array and expand when required.

2) Hope you bought all those 2TB drives before the post flood price inflation.

 

[ashrafi]

Hello! you need to format in 64k your raid and for more performance try iSCSI protocol with a gigabyte network i work with (HPC) in cluster and is the best way to work.

iSCSI with GBN ? would you explain ,
I did have latency issues within the I/O however I migrated my video server on 3TB SAS Drives , which has fixed most issues in the past ,I am using Green Drives for backup server only.

I am trying to optimizing the network throughput by GB N/W teaming , however I still face issues at the client PCs.
Any ideas how maximum n/w throughput can be achieved ?
Secondly 64k for raid , is that for video servers ( imo higher stripe size for bigger file sizes and smaller stripe for smaller files ) - thats what i know is the best practice - correct me if wrong , however having 128k can cause bottle necks ,

I recently have been looking if I can gain performance increase by enabling Write Cache .

The upgraded server is 108TB with 36*3 TB in total . 64K Raid 60 , Adaptec 6445 , SuperMicro SC847.

 

[ashrafi]

1) I hope you need the 40TB capacity now because you should always minimise the number of disks in a raid array and expand when required.

2) Hope you bought all those 2TB drives before the post flood price inflation.

How can I minimize the number of disk and expand later? once the VD has been built with Raid (5 6 60 ) the stripe data is distributed across X number of drives.

Do you mean I start Raid60 with minimum number of disk and later keep expanding or adding drives to the SAME Virtual Drive?????? I have no idea if this can be done due to parity but Would like explanation if i am wrong or u meant something else!

 

[vsyd]

Thank you!

Here are what I did

Yes, I was indeed thinking about how to built my NAS about two months. My goals are:
- GREEN (low power consumption).
- Fast so I can stream BD.ISO, MKV, DVD to my TVIX Slim S1 in my bed room and DUNE Prime in my living room.
- Connection must be Gb so I can ripped and transfer my BD.ISO in min, instead of hours or days.
- Low cost, otherwise my other significant won't approved .


Edit: add more images

Very interesting setup.

Who makes these controllers? Is there a software to manage them or are just simply managed through the front panel?

 

[FireWire2]

Thank you!

Here are what I did

Yes, I was indeed thinking about how to built my NAS about two months. My goals are:
- GREEN (low power consumption).
- Fast so I can stream BD.ISO, MKV, DVD to my TVIX Slim S1 in my bed room and DUNE Prime in my living room.
- Connection must be Gb so I can ripped and transfer my BD.ISO in min, instead of hours or days.
- Low cost, otherwise my other significant won't approved .


Edit: add more images

Very interesting setup.

Who makes these controllers? Is there a software to manage them or are just simply managed through the front panel?

DATOptic is the manufacture in US
You can manage / control it via GUI or LCD control panel - it does show SMART parameter
If you run under MAC, OpenSuSE and Windows then there are GUI app for you

 

[popatim]

really? You know better than to revive 6month old threads especially when its a two year old thread that you revived 6 months ago. Stop trying to spam this piece of crap

 

[FireWire2]

really? You know better than to revive 6month old threads especially when its a two year old thread that you revived 6 months ago. Stop trying to spam this piece of crap

Listen dude! I do not make money like to you to monitor this forum.
I only reply whenever i like or have time. Do you have problem with that?! If you do - GO POUND SAND. Stop being an A

 

[popatim]

What does of that have anything to do with you reviving old threads? hmm? You been here more than long enough to know the rules.

I'm not a moderator and I certainly dont get paid. I'm just a user who is sick of you posting about what I consider a piece of crap. Thats all you post even if its not relevant to the topic...

 

[FireWire2]

Dont be stupid! if you use and know the product and then comment, otherwise it is just plain dumb

 

[popatim]

send me 2 for review and if they are any good i'll come kiss you rear. if they are junk you come kiss mine.

 

[FireWire2]

send me 2 for review and if they are any good i'll come kiss you rear. if they are junk you come kiss mine.

You must be crazy . Why do I spend my hard earn money for your toys?!! Just stupid! If you want to check it out buy it yourself!

You may as well kiss mine now Cuz below are the review links of who actually uses the RAID
http://www.amazon.com/review/B004JPHAF0/
http://www.amazon.com/review/B004JPUZWU/
http://www.datoptic.com/ec/review/product/list/id/94/category/36/

 

[popatim]

eww.. all from people who don't know jack.
one person setup a raid 1.. .aww how cute but rather useless. What OS cant do that for you? What? Hardware raid1 you say - can certainly be done for much less than $180

Next person. 19 yrs in IT and has never setup a raid 5... need i go on?

Next person setup another raid 5 cause her motherbd raid5 was awfull. - Yet she's perfectly satisfied with her 100mb/s from this ???

next person setup 6 disks with 1 sata cable - oh, Magic is in the air. this is 5 devices and would need 6 sata cables....

Last person thought his mothebd raid was 'all or nothing' - he spent $180 and actually got less.

None of these 'clue-less' reviews even come close to making this even a fair product to recommend IMO.

- Its a port multiplier. plug 5 drives into it and plug it into your motherbd, IF it works at all you will lose 5 of your motherbd's sata ports. AFAIK Intel still doesn't support port multiplication either.
- Its bandwidth is limited to that of ONE sata port.
- This is Sata2. Max bandwidth is about 250MB/s. So much for getting any performance out of it or using it with a couple of SSD's. Intel chipset is good to almost 1000MB/s even thou it's "software raid".

Look, I'm not saying THIS is a crappy product as in: doesn't work worth beans. I don't have any experience with it. I do have experience with other Port Mult products and to me they are all crappy as in pretty useless and not worth spending any money on.

 

[FireWire2]

Next person setup another raid 5 cause her motherbd raid5 was awfull. - Yet she's perfectly satisfied with her 100mb/s from this ???

I though you are knowledgeable. Moving data speed depends on both target and source.! What about data is copying from a SINGLE HDD. Does it EVER occur to you?!
And how many users in this forum asking for help cuz their MB's RAID5 just horrible.

Next person. 19 yrs in IT and has never setup a raid 5... need i go on?

This person has the same mentality like you always refer to RAID10. lol

- Its a port multiplier. plug 5 drives into it and plug it into your motherbd, IF it works at all you will lose 5 of your motherbd's sata ports. AFAIK Intel still doesn't support port multiplication either.

- How the h3ll that you come up with this sh*t ?!! Use five SATA of your MB?!!
- When does this controller requires a SATA port supports Port Multiplier?!
It appears you have no idea and VERY low info on PM technology. FYI Intel AHCI 1.3 now supports PM natively. I bet, you don't even know that
Look at section 9.

- Its bandwidth is limited to that of ONE sata port. - This is Sata2. Max bandwidth is about 250MB/s.

Show me a hardware RAID5 cost less than $100.00, that needs no drivers and transfer about 250MB/s
Look at the video here: http://vimeo.com/29501754

So much for getting any performance out of it or using it with a couple of SSD's. Intel chipset is good to almost 1000MB/s even thou it's "software raid".

Where did you see this controller CLAIM, nor I said it will do 1000MB/s. I'm recommend mostly for NAS, Media and/or File Server with FreeNAS/OpenFiler/WHS, Home studios and use TWO (less than $200.00) used as RAID50 if there is a need for over 450MB/s transfer rate

with a couple of SSD's. Intel chipset is good to almost 1000MB/s even thou it's "software raid".

You sure does compare Orange with Orange Can you have 16TB RAID5 VOLUME ?

Look, I'm not saying THIS is a crappy product as in: doesn't work worth beans. I don't have any experience with it.

Now you admit you have NO knowledge or experience on this device, but still say it's not worth the bean, and claim it's a piece of crap?!! Because you're based on your crappy PM bridge out there?!

Let me ask you this. Intel based computers manufacture with different companies, are those systems the same or some of them are crappy as hell?
Anyway, until you use this controller. Please stop make a fool out of yourself.
I never claim this is the BEST RAID out there! All I said was there are other option, which offers decent hardware RAID for low budget, low power NAS, WHS, Media Server, even HD editing using Sony Vega and Adobe and you don't have to spend 400's for HW RAID5

My DYI NAS with SPM394 will beat ANY commercial desktop NAS out there and cost me lots less.
Do you know $1000's NAS from QNAP's and SYNOLOGIS none of them are HW raid?