Sign in with
Sign up | Sign in
Your question

40Tb Server performance issue

Last response: in Storage
Share
January 17, 2011 2:19:14 PM

I have build a media Server with following specifications,
Gigabyte GA-H55-UD3h motherboard
Processor i5 680
Dell Perc 6i Raid Card
Chenbro Expander
20 * 2Tb Western Digital Caviar Green in Raid 6 configuration
I have set Intellipark to 300secs as discussed on some forums , After installing Server 2008 R2 I transferred 3.4Tb data on the raid setup and it took 3 days . (The maximum speed reached is 30Mbps).I have set the cache block size to 512mb . I believe the speed if extremely slow , I made sure the raid card was installed in the x16 pcie slot , I might be missing something , how can I increase the performance of this media server or is it bottle neck due to hardware itself.
a b G Storage
January 17, 2011 11:47:00 PM

ashrafi said:
I have build a media Server with following specifications,
Gigabyte GA-H55-UD3h motherboard
Processor i5 680
Dell Perc 6i Raid Card
Chenbro Expander
20 * 2Tb Western Digital Caviar Green in Raid 6 configuration
I have set Intellipark to 300secs as discussed on some forums , After installing Server 2008 R2 I transferred 3.4Tb data on the raid setup and it took 3 days . (The maximum speed reached is 30Mbps).I have set the cache block size to 512mb . I believe the speed if extremely slow , I made sure the raid card was installed in the x16 pcie slot , I might be missing something , how can I increase the performance of this media server or is it bottle neck due to hardware itself.


Wow! It seems too costly

You need to see where is the problem of transfer is:

I would connect direct to another system (Window or MAC) via CAT5e cable minimum, then transfer DVD.ISO or BD.ISO files, you need to use file in range of GB

If the transfer is good then you have cable or hub/switches problem, otherwise take a close look at the system you just built, particular in network adapter property to see anything is out of ordinary

Here is the 40TB system that I built:
http://www.mpcclub.com/forum/showthread.php?t=22898&pag...



It transfers over 80MB/sec and only consume 185W max.
m
0
l
January 18, 2011 12:21:14 AM

Amazing work , Can you give me details of your build which raid and expander did you use , secondly did you change the Intelli park and TLER on these WD green drives?
I need to optimize my system so I have correct cache stride /allocation size.
I have found out that I would need some sort of block alignment , not sure about it but your example would help greatly if you took these steps .
Thanks!!
m
0
l
Related resources
a c 415 G Storage
January 18, 2011 1:07:11 AM

Write speeds will always be slow because of the need to recompute and rewrite the parity information on the array for each block that's written. But read speeds should be much faster (unless the array is still initializing or is operating in degraded mode).

With 20 drives in the array you may end up operating in degraded mode more often than you expect.
m
0
l
a b G Storage
January 18, 2011 1:58:10 AM

I have to agree, that write speed seems a little bit slow. I just configured a Dell R710 with a MD3000 disk array a few months ago, and built it as a pair of RAID 5's with a couple hot spares, and my write speeds were in the 90's. The MD3000 was full with (15) 1tb 7200rpm SAS drives, and was running on a PERC6 also. I'd check your array status. Now, of course, my speeds were gauged by copying from one internal array to the external MD3000 array. If you're copying over a LAN, then your speeds might be limited by several things. Your NIC, your switchgear, etc..... Connected with a gigabit connection, you should still be seeing transfer speeds in the neighborhood of 60MB/sec on a properly configured network (give or take, but with good switchgear and good hardware it should be in the high 70's or better)
m
0
l
a b G Storage
January 18, 2011 1:58:10 AM

Thank you!

Here are what I did

Yes, I was indeed thinking about how to built my NAS about two months. My goals are:
- GREEN (low power consumption).
- Fast so I can stream BD.ISO, MKV, DVD to my TVIX Slim S1 in my bed room and DUNE Prime in my living room.
- Connection must be Gb so I can ripped and transfer my BD.ISO in min, instead of hours or days.
- Low cost, otherwise my other significant won't approved :-).

After all research I end up with:

1x SuperMicro ITX Mobo X7SPA-O
1x Chasis http://www.norcotek.com/
4x SPM394 http://www.datoptic.com/ec/esata-hardware-raid-controll...
1x 1GB USB falsh
1x 1GB DDR2 RAM
20x WD Green 2.0TB WD20EARS
1x 430W PSU


Why X7SPA-O
I select this Mobo because, it has built in Atom CPU, TWO GbE, PCI E 4x slot and SIX (6) SATAII ports.
I do not need dual, quad core CPU since the CPU only run network services at home, and these services would not be CPU intense
so fast CPU sort of waste energy and money. Matter of fact if there is a Celeron CPU in it I would use it
- Two Gb E - I can bond (LACP) it together to get double the bandwidth 2Gbps instead of Gb Ethernet, if needed
- Empty PCI E it allows me to add 8 more SATA ports if I ever need to expand the storage so the final port I will have is twelve (12) ports - 360TB RAW volume
Or add an additional NIC to increase the bandwidth of network
- Current six SATA ports allow me to connect up to 30x SATA Drives or 90TB raw (if i use the 3.0TB HDD)
- Built in CPU - since I will not have host CPU manage the RAID so DUO Core and other CPU is wasted
- Cost in high 100's

So for who wants to build a smaller system use ITX-220 - you can have a system go up to 30TB

RAID controller SPM394
I select this because it is a stand alone hardware raid. Once the raid configured, OS sees it as ONE BIG hard drive, NO drivers required, it means I dont have to worry about update or compatible issue
It rebuilt rate is over 200GB/hr - so with a volume of 15.0TB (5x3.0TB hdd) need to rebuilt it takes 7.5Hrs or 5hrs if 5x2.0TB is used
LCD display allows me to see what is going with the array instantly
Built in alarm - sound off if there something is going wrong
No problem with 2.TB - WD20EARS - I think WD30EZRSDTL would work just fine, but I did not have this drive in hand so - test it before use

WD20EARS
Low cost
Low power - when Write / read it consume ~ 6W
Seems to work fine with SPM394, the above raid controller - there is no problem found on TLER issue



Chasis http://www.norcotek.com/item_detail....delno=RPC-4220
Low cost - rack mount decent fans
20bay - rack mount

430W PSU
As I calculate the power consumption my NAS require about 200 max, therefore 430W PSU is more than enough
20x WD = 120W max + Mobo = 45W max + 4x SPM controller @ 15W Max + Misc fan/LED... = 200W max



OS: FreeNAS
Ultimate Low cost - $0.00
USB1.0GB flash cost $5.00

My NAS Front view


Building steps:

1_ Install controller
If there is FOUR 5.25 slots that would be perfect, but there is no such thing, I decide to cut the chassis's PCI slot sopen and drop four of these SPM394 into this corner of the chassis
with couple screws I was able to hold them in place

2_ Install MoBo+Drive+ and PSU
I drop the 1GB DDR2 in 1st then mount the Mobo the chassis, after that I install 20x HDD to the tray and inserted to the chassis, the PSU is going last, the sequence I have

3_ Prepare FreeNAS flash
I re-format the existing flash as FAT
Download freeNAS CD image here and burn it to CD
Boot from just burned CD
Install FreeNAS to USB flash - for more detail you can see it at freenas.org
Plug newly loaded FreeNAS USB flash to Mobo USB port (close to SATA ports)
Just google it or drop me an email

If you need more info I will update this post

4_ Connect raid controllers to Mobo
I only use four out of six Mobo SATA ports. There are TWO empty SATA ports, this allows me to expand another Ten HDD, if i want to - This is an immediate expansion option



After it's done, the system should have four SATA ports connected to four SPM394 and FOUR SPM394 SATA ports fan out to 20x WD HDD. An USB flash with freeNAS is connected to onboard USB port



(Continue...)

Edit: add more images
m
0
l
a b G Storage
January 18, 2011 5:32:43 PM

Part II

5_ Set up NAS

- Connect a keyboard and monitor - only for set up, after that system can run headless
- Power the system, go to BIOS change these:
- SATA II mode: Enhance or AHCI
- Boot drive: USB flash
- Disable Audio, Parallel port
Save setting and boot
Once it powered, configure four raid5 out of SPM394 via LCD panel, the end result Mobo will see 4x 8.0TB HDD, then we combine those to ONE big 32TB volume as RAID50

I'm configured this way, because the rebuilt time is minimal and one parity per five drive is lots more reliable then two parity for 20 drives (raid6).

I did a test on this RAID50 in my Win7 system, I get 840MB/sec, so the speed from the RAID is not a problem that I have to worry.

6_ Once it booted, the default IP address is 192.168.1.250. Using the connected KB change the IP to the existing network address

7_ Open Web Browser in your MAC/Window system, enter the new configured IP. Your should be able to log-in and configure the rest
With a little tune-up in SMB protocols, the system can easily transfer @ 80MB/sec or more



There are many features in FreeNAS, that I found useful for the home user:
Rsync - Remote backup MAC/Window data
iSCSI target - you can share the storage to your system
Bittorrent - Download your favorite media content


If you need help or have question, please post here, I will try my best to answer

FireWire
m
0
l
January 25, 2011 11:52:49 AM

Nice post , In my honest opinion you could have saved lots of money if you used Expander card and connected External bays through it , I have not used this specific controller therefore I am unaware of features it offers , I was confused on how you individually configured Raid 5 (8 Tb ) initially and finally it was configured as 32Tb single volume , ( was it all controller based ? ..I mean i know how raid controllers have their bios and everything and it is easily configured ) .

Secondly , about inserting USB drive on the mobo , can you explain how you did that , ( i assume it always boots by default to the USB ).

LACP bonding , never experienced it myself , can you explain how your achieved it ?

"With a little tune-up in SMB protocols, the system can easily transfer @ 80MB/sec or more"..What tuneups did you do and how?

overall your setup is impressive , but while i read your post i wondered if this motherboard had the capacity to hold all this bandwidth ? I was trying to convince myself the upper $200 price for this motherboard would justify .
I ask this because earlier I had experience using Zotac Mini ITX for a small Storage-Server (10 TB ) , this specific motherboard had 6 sata ports as well , ( Processor price was def add up but I wanted to use decent i5 , further this mobo provided wifi , hdmi , Spdif)
m
0
l
a b G Storage
January 25, 2011 10:41:46 PM

Thank you

I'm not sure about expander that you're refer to. As far as i know, there are TWO type expanders: PM (Port Multiplier) and SAS expander.
These are just simple an one to many adapters, both does not offer raid function built in
- PM bridge is a 1 to 5 - there is NO hardware raid controller that supported PM, except SPM394
- SAS expander is a 1 to 16/24 ports - too expensive because to get it works you need SAS raid + SAS expander that would be over 1000's dollars

so for the cost i used SPM394 or SPM393. I have to configure 4x raid5 and combine it, because the SATA ports in Mother board are NOT Port multiplier supported, it SEES A SINGLE volume/drive only. All five drives configured as RAID5 in SPM394, it is treated as a SINGE drive by SATA host controller.

In other world I want five HDD through SPM394 shows up as a SINGLE drive in freeNAS, where raid5 function reliability and performance will handle by the SPM394 independently from FreeNAS.

There are pros and cons on my configuration
Pro: Low cost, easy maintenance, fast rebuilt. Most important there is no drivers or additional software needed, so if i update NAS firmware these raid arrays will not give me any compatible issue due to drivers like other raid card does

Con: loose 4 drives capacity in stead of 1 (8TB instead of 2TB), customize mounting these controllers

Note: Since SPM394 controls 5x HDD and presents to SATA host port as 8TB HDD, you need to make sure that SATA host port is not 2.0TB limited, don't be surprise if your board does, there are many SATA port that CAN NOT see a HDD over 2.0TB

Quote:
Second...

After loaded freeNAS into USB flash, just connected it to USB port. Set up the BIOS to boot up from it - that is - this USB is an OS drive, like C drive in Windows
Once powered the OS (freeNAS) will be loaded run. there is another benefit that I have not mention, when runs freeNAS potential virus thread is minimized.
you can download this manual http://www.supermicro.com/manuals/motherboard/ICH9/MNL-.... Search for "JUSB5", It will show you the connection

Quote:
LACP...
please refer to: http://en.wikipedia.org/wiki/Link_Aggregation_Control_P...

Quote:
i wondered if this motherboard had the capacity to hold all this bandwidth ?

Yes it is, this board has an Intel I/O controller Hub ICH9R with GB of bandwidth internal and dual PCI express Intel NICs (one of the BEST NIC) built-in, I don't think you can get a better mobo for NAS than this board. Using Intel NIC alone, you would has 30% more transfer rate compare to Realtek NIC.

Of course you can use any the Mobo, but one use with FreeNAS, the HDMI, SPDIF, HiFi Audio, Wifi feature are not used. Matter of fact I disable everything in the BIOS: COM ports, Printer port, HD Audio...

Just keep in mind, the way I configure the raid array DOES NOT using Mother board CPU, the MAIN task for CPU is manage the network services, and this does not need a whole lot power :-). There is CPU graph in freeNAS that allows you to see CPU loading in real time. I have not able to load it to 100% on this Intel® Atom™ D510 processor .

Also when you using NAS
Quote:
With a little tune-up in SMB protocols, the system can easily transfer @ 80MB/sec or more...


reading various post in internet - Here is my TUNE UP on our NAS:

1- Change Swap configured as: Type=File / Mount Point=Hardware Raid Volume / Size=512M
2- System | Advanced| sysctl.conf
- net.inet.tcp.recvspace = 524288
- net.inet.tcp.sendspace = 524288
- net.inet.tcp.delayed_ack=0

3 - Network | LAN Management
- MTU=9000
- Device polling=Checked

4 - Services|CIFS/SMB|Settings
- Large read/write = Checked
- Asynchronous I/O (AIO)=checked
- Auxiliary parameters:
deadtime = 1
getwd cache = yes
max xmit = 65535 # 32768 # 65535
read raw = yes
socket options = TCP_NODELAY
strict locking = no
write raw =yes

With these change I get over 80MB/sec

For whom who may not need 20bays system - look at this eBOX-N, this can handle 30TB (10x 3.0TB)

FireWire
m
0
l
May 27, 2011 12:17:20 AM

Superb Post, I am hoping you can clarify


FireWire2 wrote:

"Once it powered, configure four raid5 out of SPM394 via LCD panel, the end result Mobo will see 4x 8.0TB HDD, then we combine those to ONE big 32TB volume as RAID50"

ashrafi ask:

"I was confused on how you individually configured Raid 5 (8 Tb ) initially and finally it was configured as 32TBb single volume , ( was it all controller based ? ..I mean i know how raid controllers have their bios and everything and it is easily configured ) ."

I am guessing that you used the LCD control panel of each SPM394 Raid Controller to format, initialize and create a five disk Raid 5 Array on each SM394 controller. Is that correct ???

I am also guessing that you used the bios of your mother board to combine the four Raid 5 Arrays - with each array appearing as a single disk to the bios - into a "4 disk" Raid 0 array using the bios on the Mother Board to create the Raid 0 array, is this correct ???

I am now guessing the Raid 0 array now appears as a single 32TB hard disk to the operating system on the NAS computer you created, is this
correct ???

I am also guessing your reference to a Raid 50 array meant a Mother Board Bios Based raid 0 built on top of 4 independent SPM394 controller based Raid 5 arrays, is that correct ???

Again, great post.
m
0
l
a b G Storage
May 27, 2011 10:22:41 PM

Cost
If I use PCI Express it costs me at least $1450.00 for 20+ SATA port (1 year ago)
where my solution cost me merely $860.00 - near $600.00 saving where i can spend on HDD

Easy
With PCI express, I has to use CLI (command line interface), and not quite sure about loading the drivers in freeNAS

Where SPM394, which will group 5 drives as RAID5 (data protected) and present it as a SINGLE drive to the BIOS. Which is nice because there is no driver, software to install or messing around
Addition to:
a) MB SATA port does not support PM ware so there is no way I can see 20 drives in FreeNAS
b) Assume I can see 20x drive in and created a parity raid (raid5/6) - It will put a big dent to host CPU resource

So SPM394 is a best/simplest solution for my need

Here are the answer to your questions:

Quote:
I am guessing that you ....

- Yes! with a couple button press I have 8TB volume under a minute

Quote:
I am also guessing that ....

- No, i do not use the MB raid function because freeNAS not supported all MB raid, I just use it as AHCI, so FreeNAS will see four (4) 8TB drives.

In FreeNAS Disk Manger, then use JBOD* or RAID0 to combine it to 32TB volume.

Quote:
Reliable

So far it runs 24/7 about 1 yr.


* this should be name as BIG or SPAN to be correct




m
0
l
December 1, 2011 7:20:20 PM

Hello! you need to format in 64k your raid and for more performance try iSCSI protocol with a gigabyte network i work with (HPC) in cluster and is the best way to work.
m
0
l
a b G Storage
December 2, 2011 12:06:45 PM

1) I hope you need the 40TB capacity now because you should always minimise the number of disks in a raid array and expand when required.

2) Hope you bought all those 2TB drives before the post flood price inflation.
m
0
l
December 9, 2011 2:46:57 AM

Cracktroz said:
Hello! you need to format in 64k your raid and for more performance try iSCSI protocol with a gigabyte network i work with (HPC) in cluster and is the best way to work.


iSCSI with GBN ? would you explain ,
I did have latency issues within the I/O however I migrated my video server on 3TB SAS Drives , which has fixed most issues in the past ,I am using Green Drives for backup server only.

I am trying to optimizing the network throughput by GB N/W teaming , however I still face issues at the client PCs.
Any ideas how maximum n/w throughput can be achieved ?
Secondly 64k for raid , is that for video servers ( imo higher stripe size for bigger file sizes and smaller stripe for smaller files ) - thats what i know is the best practice - correct me if wrong , however having 128k can cause bottle necks ,

I recently have been looking if I can gain performance increase by enabling Write Cache .

The upgraded server is 108TB with 36*3 TB in total . 64K Raid 60 , Adaptec 6445 , SuperMicro SC847.





m
0
l
December 9, 2011 1:42:42 PM

Rusting In Peace said:
1) I hope you need the 40TB capacity now because you should always minimise the number of disks in a raid array and expand when required.

2) Hope you bought all those 2TB drives before the post flood price inflation.



How can I minimize the number of disk and expand later? once the VD has been built with Raid (5 6 60 ) the stripe data is distributed across X number of drives.

Do you mean I start Raid60 with minimum number of disk and later keep expanding or adding drives to the SAME Virtual Drive?????? I have no idea if this can be done due to parity but Would like explanation if i am wrong or u meant something else!
m
0
l
February 13, 2013 11:09:32 PM

FireWire2 said:
Thank you!

Here are what I did

Yes, I was indeed thinking about how to built my NAS about two months. My goals are:
- GREEN (low power consumption).
- Fast so I can stream BD.ISO, MKV, DVD to my TVIX Slim S1 in my bed room and DUNE Prime in my living room.
- Connection must be Gb so I can ripped and transfer my BD.ISO in min, instead of hours or days.
- Low cost, otherwise my other significant won't approved :-).


Edit: add more images


Very interesting setup.

Who makes these controllers? Is there a software to manage them or are just simply managed through the front panel?
m
0
l
a b G Storage
August 12, 2013 6:44:39 PM

vsyd said:
FireWire2 said:
Thank you!

Here are what I did

Yes, I was indeed thinking about how to built my NAS about two months. My goals are:
- GREEN (low power consumption).
- Fast so I can stream BD.ISO, MKV, DVD to my TVIX Slim S1 in my bed room and DUNE Prime in my living room.
- Connection must be Gb so I can ripped and transfer my BD.ISO in min, instead of hours or days.
- Low cost, otherwise my other significant won't approved :-).


Edit: add more images


Very interesting setup.

Who makes these controllers? Is there a software to manage them or are just simply managed through the front panel?


DATOptic is the manufacture in US
You can manage / control it via GUI or LCD control panel - it does show SMART parameter
If you run under MAC, OpenSuSE and Windows then there are GUI app for you
m
0
l
a c 885 G Storage
August 13, 2013 4:47:02 PM

really? You know better than to revive 6month old threads especially when its a two year old thread that you revived 6 months ago. Stop trying to spam this piece of crap
m
0
l
a b G Storage
August 14, 2013 11:22:33 AM

popatim said:
really? You know better than to revive 6month old threads especially when its a two year old thread that you revived 6 months ago. Stop trying to spam this piece of crap


Listen dude! I do not make money like to you to monitor this forum.
I only reply whenever i like or have time. Do you have problem with that?! If you do - GO POUND SAND. Stop being an A
m
0
l
a c 885 G Storage
August 14, 2013 1:05:05 PM

What does of that have anything to do with you reviving old threads? hmm? You been here more than long enough to know the rules.

I'm not a moderator and I certainly dont get paid. I'm just a user who is sick of you posting about what I consider a piece of crap. Thats all you post even if its not relevant to the topic...
m
0
l
a b G Storage
August 14, 2013 3:12:04 PM

Dont be stupid! if you use and know the product and then comment, otherwise it is just plain dumb

m
0
l
a c 885 G Storage
August 14, 2013 5:15:34 PM

send me 2 for review and if they are any good i'll come kiss you rear. if they are junk you come kiss mine.

m
0
l
a b G Storage
August 21, 2013 12:34:03 PM

popatim said:
send me 2 for review and if they are any good i'll come kiss you rear. if they are junk you come kiss mine.


You must be crazy :no: . Why do I spend my hard earn money for your toys?!! Just stupid! If you want to check it out buy it yourself!

You may as well kiss mine now :-) Cuz below are the review links of who actually uses the RAID
http://www.amazon.com/review/B004JPHAF0/
http://www.amazon.com/review/B004JPUZWU/
http://www.datoptic.com/ec/review/product/list/id/94/ca...

m
0
l
a c 885 G Storage
August 21, 2013 3:44:28 PM

eww.. all from people who don't know jack.
one person setup a raid 1.. .aww how cute but rather useless. What OS cant do that for you? What? Hardware raid1 you say - can certainly be done for much less than $180

Next person. 19 yrs in IT and has never setup a raid 5... need i go on?

Next person setup another raid 5 cause her motherbd raid5 was awfull. - Yet she's perfectly satisfied with her 100mb/s from this ???

next person setup 6 disks with 1 sata cable - oh, Magic is in the air. this is 5 devices and would need 6 sata cables....

Last person thought his mothebd raid was 'all or nothing' - he spent $180 and actually got less.

None of these 'clue-less' reviews even come close to making this even a fair product to recommend IMO.

- Its a port multiplier. plug 5 drives into it and plug it into your motherbd, IF it works at all you will lose 5 of your motherbd's sata ports. AFAIK Intel still doesn't support port multiplication either.
- Its bandwidth is limited to that of ONE sata port.
- This is Sata2. Max bandwidth is about 250MB/s. So much for getting any performance out of it or using it with a couple of SSD's. Intel chipset is good to almost 1000MB/s even thou it's "software raid".

Look, I'm not saying THIS is a crappy product as in: doesn't work worth beans. I don't have any experience with it. I do have experience with other Port Mult products and to me they are all crappy as in pretty useless and not worth spending any money on.
m
0
l
a b G Storage
August 21, 2013 8:16:26 PM

popatim said:
Next person setup another raid 5 cause her motherbd raid5 was awfull. - Yet she's perfectly satisfied with her 100mb/s from this ???

I though you are knowledgeable. Moving data speed depends on both target and source.! What about data is copying from a SINGLE HDD. Does it EVER occur to you?!
And how many users in this forum asking for help cuz their MB's RAID5 just horrible.

Quote:
Next person. 19 yrs in IT and has never setup a raid 5... need i go on?

This person has the same mentality like you always refer to RAID10. lol

Quote:
- Its a port multiplier. plug 5 drives into it and plug it into your motherbd, IF it works at all you will lose 5 of your motherbd's sata ports. AFAIK Intel still doesn't support port multiplication either.

- How the h3ll that you come up with this sh*t ?!! Use five SATA of your MB?!! :pt1cable: 
- When does this controller requires a SATA port supports Port Multiplier?!
It appears you have no idea and VERY low info on PM technology. FYI Intel AHCI 1.3 now supports PM natively. I bet, you don't even know that :-)
Look at section 9.

Quote:
- Its bandwidth is limited to that of ONE sata port. - This is Sata2. Max bandwidth is about 250MB/s.

Show me a hardware RAID5 cost less than $100.00, that needs no drivers and transfer about 250MB/s
Look at the video here: http://vimeo.com/29501754

Quote:
So much for getting any performance out of it or using it with a couple of SSD's. Intel chipset is good to almost 1000MB/s even thou it's "software raid".


Where did you see this controller CLAIM, nor I said it will do 1000MB/s. I'm recommend mostly for NAS, Media and/or File Server with FreeNAS/OpenFiler/WHS, Home studios and use TWO (less than $200.00) used as RAID50 if there is a need for over 450MB/s transfer rate

Quote:
with a couple of SSD's. Intel chipset is good to almost 1000MB/s even thou it's "software raid".

You sure does compare Orange with Orange :-) Can you have 16TB RAID5 VOLUME ?

Quote:
Look, I'm not saying THIS is a crappy product as in: doesn't work worth beans. I don't have any experience with it.

Now you admit you have NO knowledge or experience on this device, but still say it's not worth the bean, and claim it's a piece of crap?!! Because you're based on your crappy PM bridge out there?!

Let me ask you this. Intel based computers manufacture with different companies, are those systems the same or some of them are crappy as hell?
Anyway, until you use this controller. Please stop make a fool out of yourself.
I never claim this is the BEST RAID out there! All I said was there are other option, which offers decent hardware RAID for low budget, low power NAS, WHS, Media Server, even HD editing using Sony Vega and Adobe and you don't have to spend 400's for HW RAID5

My DYI NAS with SPM394 will beat ANY commercial desktop NAS out there and cost me lots less.
Do you know $1000's NAS from QNAP's and SYNOLOGIS none of them are HW raid?


m
0
l
November 7, 2013 2:29:23 PM

I agree with Firewire2. That is a seriously GREEN nas, and it is very fast if configured correctly.

I have something similar myself, but it is 20Tb Raid 50 instead of 40Tb. So far it has been up for 193 days without issue. No drives have failed yet so I can't comment on the rebuild procedure but hopefully it should be straight forward.

Keep in mind that the Freenas implementation of Samba is single-threaded, so to get maximum single user performance it is better to use multi-threaded NFS or maybe iSCSI.

Also, to be fair, this is an old thread now and Freenas has come along way. I also have a 30Tb ZFS Freenas server and it is very impressive!

With ZFS you don't need a raid controller ($) , but you will need a good mainboard, fast cpu, and lots of ram to get fast transfer speed.

Both solutions have pros and cons, I would recommend either to anyone looking into serious DIY storage.

Firewire, do you like ZFS?



m
0
l
a b G Storage
November 7, 2013 5:37:14 PM

On ZFS, i need to learn more.
What is your hardware setup for 30TB ZFS, I'm thinking build a Universal Media Server to stream on-fly encoding media content to any DLNA, UPnP devices. Your set up can give me some starting point

Thanks
m
0
l
November 8, 2013 9:02:31 AM

Here is the 30TB DIY ZFS server and an outline of why this hardware was chosen.

Firewire's Nas is green and very fast. For this 30TB ZFS server, we needed full disk encryption as it is being used for off-site backup. The Atom processor is not really up to the job of encryption or ZFS, so I looked into using more powerful hardware and decided on the following:

Specification

1 x Supermicro H8SML-7F motherboard
1 x AMD FX-6300 CPU
4 x Kingston 8GB ECC RAM ( KVR16E11/8 )
10 x Seagate 3TB Barracuda Hard disks ( ST3000DM001 )
1 x 8gb USB memory stick
1 x 450w PSU
1 x Xigmatek elysium tower case ( You can use a rack case with at least 10 sata bays if you prefer )

With this specification, a lot of money is saved by not buying a raid controller, but is spent on the high-end motherboard, powerful CPU, and lots of RAM.

H8SML-7F motherboard

I chose this motherboard because it has 6 onboard SATA and 8 onboard SAS connections. As SATA is a subset of sas we can use SATA drives without issue. The onboard SAS connections are individual so we use standard SATA cables to connect the drives to the mainboard.

Also this mainboard has dual onboard intel gigabit lan, which means 2gb link saturation if you use good configuration, cables, and switches.

AMD FX-6300 cpu

The Mainboard supports socket AM3+ opteron CPU's and works fine with this much cheaper and excellent value socket AM3+ desktop processor. This CPU is very powerful and has AES-NI instruction set. Freenas recognizes this as 'crypto hardware' which means that the full disk encryption performance hit is negligible.

4 x Kingston 8GB ECC RAM

With ZFS you should use as much RAM as possible for the ARC cache, but the standard recommendation is 1GB of RAM per 1TB of storage. So the 32GB of RAM in this server is enough for the 30TB raw storage.

10 x Seagate 3TB Barracuda Hard disks

These are cheap high performance consumer disks. If power usage is a concern then Western Digital green disks are better.

1 x 8gb USB memory stick

To use as a Freenas boot drive

1 x 450w PSU

More then enough power for this server.

1 x Xigmatek elysium tower case

I used this case because for this site a tower is better then a rack case. Any case with at least 10 sata drive bays is ok. For example, the Norco RPC-2212 12 bay 2u case with SATA backplane option.

Configuration

Once the server is built and Freenas is installed it is configured to use full disk AES encryption and RAIDZ2 ( broadly similar to RAID 6 ). This gives a single logical volume with 2 disk redundancy.

This is very easy to configure with Freenas and all you need to do is tick a few boxes!

ZFS is clever, and automatically places the most active files into the ARC cache, which means lower disk wear and higher performance, If you like you can even use SSD as ARC cache.

I am only using 1gb lan for transfers and it averages around 90MB/s. When using the ZFS ARC Cache it can sustain 125MB/s.

ZFS vs Hardware RAID

ZFS is very advanced and is definitely worth learning about. The 30TB DIY ZFS NAS I have built is not as fast as my 20TB SPM394 RAID50 but in theory is slightly more reliable as any 2 disks can fail as opposed to 1 out of 5 disks with the SPM394.

The encrypted ZFS Freenas rebuild procedure looks complicated and will probably involve some CLI. This is not a problem for me, but for ease of use it is better to use a hardware RAID controller like SPM394 with auto rebuild.

Any questions please ask.



m
0
l
a b G Storage
November 8, 2013 1:38:53 PM

Wow! I'm impress :-) Excellent post.

Thank you very much - now back to the purchasing cycle
m
0
l
a c 885 G Storage
November 8, 2013 3:49:04 PM

Sounds like a very nice server you have built. How do you back all that up?
m
0
l
August 7, 2014 9:30:10 AM

FireWire2 said:
Wow! I'm impress :-) Excellent post.

Thank you very much - now back to the purchasing cycle



Firewire are you still using the SPM394. I am thinking about buying one, I had a RAID 5 in an external enclosure with a Highpoint Rocketraid 622 card and just lost everything during a rebuild, comp restarted in middle of night for an update so no I am looking for a different solution, I am thinking of building a freenas unit with the SPM394 or is there a better option?
m
0
l
a b G Storage
August 7, 2014 9:56:42 AM

SPM394 is perfect solution for FreeNAS, every single NAS that i did has SPM394.
Note: SPM394 can transfer over 230MB, so even with dual NICs bonding, you still OK...
m
0
l
August 19, 2014 9:39:18 AM

FireWire2, having as an OS Windows based system (win7 or win srv 2012) can I use 2-3 SPMs394, build 3 x RAID5 arrays and then "merge" then into one big volume, as you have described with freeNAS?
I am asking because I have an app that runs only on windows systems.
m
0
l
August 19, 2014 11:56:49 AM

qhash said:
FireWire2, having as an OS Windows based system (win7 or win srv 2012) can I use 2-3 SPMs394, build 3 x RAID5 arrays and then "merge" then into one big volume, as you have described with freeNAS?
I am asking because I have an app that runs only on windows systems.


That is exactly what i am doing.

I merged the SPM394 arrays as a "dynamic disc". So far uptime over 1 year no problems.
m
0
l
a b G Storage
August 19, 2014 12:17:23 PM

With 3x SPM394 set as "dynamic disc"like Storage_Enthusiast, and striping it instead merage (SPAN), you're not only get 3x times as big single volume, but 2.5x faster transfer rate.
Typical SPM394 read/write @ 235~260MB/s, with RAID50 as above you would have 650~ 725MB/s
m
0
l
August 19, 2014 1:23:38 PM

OK but if somehow one of the striped RAID5 elements fails (it can be second drive during rebuild, the SPM394 itsefl), then i lose all the data across the dynamic volume? I do not need more than 50-60MB/s. I need to be rock solid.
m
0
l
a b G Storage
August 19, 2014 1:32:14 PM

I guess that can happen, then SPAN mode is the ideal for you
m
0
l
a c 885 G Storage
August 19, 2014 6:42:04 PM

Even with the spm modules, you will still be in the same boat as before - lose a drive and something/anything happens during the rebuild its *poof* bye- bye data again.

Perhaps you should consider mirroring the arrays instead. or maybe a raid 10
m
0
l
August 19, 2014 10:59:11 PM

RAID 10 is perfect, but it means losing half of the drives... maybe if I buy that norco rack, put 20 cheaper drives, it will fit the budget. Oh, if we are touch ing the rack chassis with hot-swap 3,5 bays - is there something smaller and cheaper that this norco case - for 12 or 16 drives?
m
0
l
August 20, 2014 4:33:07 AM

"RAID 10 is perfect, but it means losing half of the drives"

I haven't tried this myself but if you are feeling bold you could, I think, build a freenas system with ZFS RaidZ2 like I detailed earlier in this post and run windows as a virtual machine (Jail). Then, for example, you could lose 2 out of 10 drives as opposed to 1 out of 5 drives.

m
0
l
a b G Storage
12 minutes ago

RAID10 to costly... I would merge multiple SPM394/393 into a BIG Volume for production, then use a LOW COST Green drive with PM to create another volume for back up...
Using SPM394/393 offer one drive protect for every 5 drives, SPAN volume use as back up in case.
m
0
l
!