8-bay+ DIY raid solution for video production

[BrodyJ]

Hey guys,

I'd like to build a DIY raid solution for video production. I did a lot of research but haven't been able to find something out there exactly like I'd want to build.

Here are the details:
1. I want to build a separate machine for this raid that will be in the equipment room.
2. I'd like to make it rack mountable
3. I'm aiming for 8 bays or possibly more
4. Support for 4TB drives
5. I'm planning to use RAID5
6. I'd like to match or exceed speeds people are getting from the Promise Pegasus (500+mb/s read/write in Raid 0  + 300mb/s read and 500mb/s write in Raid5)
7. Easy to replace disks in case of failure.
8. The machine the raid will be connected to will be a Mac.
9. Ideally a hardware raid (unless software could be as good - see below)
10. I'm looking for something that's reliable, stable, fast, affordable and with low maintenance.


I'd like to do this under $1000, is that possible? (excluding the drives)
My head is spinning for instance because it seems there are so many little details to consider and the choice of hardware is more difficult than I thought. e.g. there is such a huge range between raid cards (from $100 to $1000 and up). In my mind this would as simple as getting an enclosure (like a server chassis), a mobo + processor, memory, raid card and some other hardware and you're off to go. But it seems it's not just a matter of getting the right hardware together. Or perhaps it is and I haven't been reading the right stuff?

Also there's the software raid vs. hardware raid topic.
Since I'm building a separate machine for this the issues of adding load to the CPU are moot. Are there others concerns to take into account with a SW raid? Although at first I was going 100% for a hardware raid solution, if I can get a software solution answering to all of my requirements listed above, then I'd consider it.

What would you suggest?
Would be really grateful for your help.
Brody

 

[FireWire2]

Hey guys, ....

Under $1000?!! Yes possible, but wont getting 700MB/s.
Here are the solutions for both cost over adn under 1K

1_ > $1000 - RAID system
Get 2x SPM393 - eSATA controller hardware raid - cost you about $200.00
http://www.amazon.com/gp/product/B005G5L98G
Get 10x HDD whatever size you want.
Get a PCIe eSATA card at least 2x port with x4 or 8x lanes like Rocket 644L (not RAID card) - costs about $80.00
http://www.amazon.com/HighPoint-ROCKET-External-Expressx4-Controller/dp/B00A6O9PIS
http://www.newegg.com/Product/Product.aspx?Item=N82E16816115119

Create TWO RAID5 out of 10x HDD with SPM393-E
Connect these TWO RAID5 to Rocket 644L via 2x eSATA cable (6Gb/s bandwith)
Create RAID0 from DU or 644L

Now you have a RAID50 which can transfer about 450MB/s ~ 480MB/s R/W

2_ > $1000 - RAID system - will get 550MB~850MB/s transfer rate
Get a Dual miniSAS HW raid card from ARECA, HPT, ATTO, ADAPTEC, LSI costs about $800 ~ $1000
Get two miniSAS cables
Get two miniSAS to 4x SATA module
Get two 6G/s Tray-less SATA drive cage - Note: there are many of 3Gb/s tray-less out there so be careful

If you have a PCIe V2 system then you can get about 700MB~850MB/s - same as TBolt
but if the system only offer PCIe V1, then 550~700MB

Both solutions offer hot swap HDD - for easy rebuild, replace BAD HDD

If you need more detail - let me know

 

[popatim]

Brody, how will you be connecting the Mac to this array you wish to build?

 

[choucove]

My first thought would be to set up a rackmount DAS (Direct Attached Storage) utilizing miniSAS 6 Gbps connections. There are a wide range of rackmount DAS chassis, all you have to do is add your hard drive storage and your PCI-Express RAID controller with external miniSAS ports. However, since you are using a Mac for hosting this storage array, I'm not sure you will have driver support for many of those RAID controllers. It's going to be a matter of tracking down just the right card to meet your compatibility needs, yes you can set up a DAS for around $1,000, but without investing in the right high-performance RAID controller, enclosure, and hard drives, then you're probably not going to be able to push that kind of throughput in RAID 5. A RAID 10 array yes, but not RAID 5.

 

[BrodyJ]

Hey guys. Very grateful for your help. Sorry for my delay in answering, I've been away from home and hope this thread can still remain alive Very excited about this project.

So in Firewire2's and choucove's suggestions it basically ends up being a raid controller, some simple SAS & sata connectors/hardware and that's it.

So you mean I could just get a box with a PSU and cooling fans and a RAID card connected to a PC/Mac and I wouldn't need to build a second computer just for the raid system?
And we're talking hardware RAID here so it wouldn't overload the CPU or anything like that?… Interesting.

In all of the research and builds I've seen up to now it seemed that in order to build a separate RAID system it would need to be a mini computer on its own and wouldn't be the same as building a RAID inside a computer.
Am I getting this right? I don't need a mobo, memory and everything? (that's just for software raids?)


Firewire2 Thanks a lot for your suggestions/builds.

Concerning Option 1
So I could throw that in any sort of case and I'd be good to go?
How would I be connecting this to my Mac?

Option 2
Concerning option 1 vs option 2, which solution would answer point 10 of the original post the best > most reliable, stable, and low maintenance + ease to set it up.

What about this card? The HighPoint RocketRAID 2720SGL SATA 6G RAID Controller. Apprently for its price nothing beats it (see link) and it appears I would get the speed I want. Am I missing anything?

Some questions / notes
- I'd be willing to buy more expensive cards but I would need to know what I would gain from doing so, i.e. if there is a marginal difference in speed, reliability or ease of use. Specifically in regards to the inexpensive card linked above.
- Are the speeds you shared 8-bay+ Raid5 speeds?
- Can I set up these build suggestions relatively easily via a Mac? (all the builds I've seen to date using PCs and were tweaking stuff in DOS and whatnot) Since your propositions don't talk about building a PC for the raid system, then I would need to set this up on the Mac side, or if need be, via Windows running on my Mac.

 

[BrodyJ]

Brody, how will you be connecting the Mac to this array you wish to build?

Well that was one of the things I needed to resolve. I'm looking for the fastest I/O available so in my mind there was only Fibre to consider. I'm also building myself a Mac from scratch so I need to figure out what card would fit the bill. There is a HUGE range of prices for these 6gb/s fibre cards so I guess I would need guidance with that eventually

 

[choucove]

What I am kind of recommending is a RAID subsystem sort of like this:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816111449

There are a lot of different brands, options, etc. but this one in the link includes the necessary RAID controller as well to attach to the DAS unit. For the greatest performance I would probably recommend going with a different brand and use a different RAID controller as well, but basically you need a PCI-Express RAID controller with external miniSAS connection. This is going to offer you SAS speed directly accessed through your physical machine for large array capacities. However, I can't find information on compatibility with OSX, which I'm going to imagine there is none. OSX is very locked down and limited in driver support for this kind of extra hardware, though you might be able to find workarounds.

 

[popatim]

Brody, how will you be connecting the Mac to this array you wish to build?

Well that was one of the things I needed to resolve. I'm looking for the fastest I/O available so in my mind there was only Fibre to consider. I'm also building myself a Mac from scratch so I need to figure out what card would fit the bill. There is a HUGE range of prices for these 6gb/s fibre cards so I guess I would need guidance with that eventually

Since you seemed to want to put this in another room entirely I was afraid you hadn 't considered that aspect. It'd be pointless to build something like that and run it on a gigabit lan. Glad to see you have a good head on your shoulders.

 

[BrodyJ]

So how would a PC machine connect to raid solution like we're talking here? I assume you need a card with SAS inputs that you put in your user/work machine?
Then I guess I would need to find if there are compatible Mac options.

Concerning the enclosure, I've been looking at those (and more bays rack solution). Pretty affordable and unless I'm missing something I could just throw in a raid controller and SAS connections (if compatible with Macs) and would be good to go?
http://www.sansdigital.com/towerraid-/tr8xplus.html

 

[FireWire2]

So how would a PC machine connect to raid solution like we're talking here? I assume you need a card with SAS inputs that you put in your user/work machine? ...
http://www.sansdigital.com/towerraid-/tr8xplus.html

If your system is genuine MAC, considered TB solution, all new MACs now offer TBolt. Keep in mind SAS to Tbolt is very expensive

But if your system is Hackintosh, then SAS controller would give you the BEST performance and cost not as high as TBolt.

Certainly you can use TR8x with ANY SAS card. But RR2720 is not a HW raid, RAID calculation will handle by host CPU.

 

[choucove]

So how would a PC machine connect to raid solution like we're talking here? I assume you need a card with SAS inputs that you put in your user/work machine?
Then I guess I would need to find if there are compatible Mac options.

Concerning the enclosure, I've been looking at those (and more bays rack solution). Pretty affordable and unless I'm missing something I could just throw in a raid controller and SAS connections (if compatible with Macs) and would be good to go?
http://www.sansdigital.com/towerraid-/tr8xplus.html

Yes that's basically what you need for this configuration is a RAID controller with external miniSAS connection and then the actual SAS cable to connect to the external RAID subsystem. Because it is SAS 6 Gbps compatible, you have up to 6 Gbps of throughput read AND write which would give you the throughput performance you are looking for with the right drives in RAID 10/5. Finding a Mac compatible RAID controller and driver is going to be your real issue, though. I have not done it before, but supposedly if you didn't have RAID drivers for a card but did have drivers which at least recognized the hard drives, you could use an HBA (host bust adapter) SAS card instead, and use the software within your OS to set up the RAID volume, but that's pretty iffy to me.

 

[FireWire2]

Hey guys. Very grateful for your help. Sorry for my delay in answering, I've been away from home and hope this thread can still remain alive Very excited about this project.
....
How would I be connecting this to my Mac?

Option1
The simplest solution is SPM394/SPM393, just plug and play, there is NO software or drivers need to install, this mean less compatible issue, but you wont get 700MB/s out of it
ANY case with PSU 300W would do, but I refer this case:
http://www.microcenter.com/product/393941/Prodigy_Mini-ITX_Computer_Case_-_Black

If you have MACPro and NO eSATA then get 3.0Gb or 6.0Gb eSATA PCIexpress like this

If you have MACPro 2010 and earlier models use the build-in SATA port with eSATA bracket

If you have a Hackingtosh, then use ANY SATA port with eSATA bracket. That will work.

For performance, use a tru HW raid, do not use middleware RAID --> RR2710, it use host CPU, and rebuild rate is terrible. It takes days to rebuild.
Use this: http://www.amazon.com/Sans-Digital-HA-ARC-ARC1882X-Mini-SAS-Controller/dp/B007B5Q1GY

 

[Snowroute Towzone]

So how would a PC machine connect to raid solution like we're talking here? I assume you need a card with SAS inputs that you put in your user/work machine?
Then I guess I would need to find if there are compatible Mac options.

Concerning the enclosure, I've been looking at those (and more bays rack solution). Pretty affordable and unless I'm missing something I could just throw in a raid controller and SAS connections (if compatible with Macs) and would be good to go?
http://www.sansdigital.com/towerraid-/tr8xplus.html

I may be a little late to the party, but here's a completely different line of reasoning. If you're building your own Hackintosh, you can build it using a MOBO with Thunderbolt connections. I know, as I recently have done so. I also want fast storage for video, and have looked into a DIY RAID solution (RAID 5 or 10, or 50) The limiting factor on the Mac side is the controller, both from the price and the hardware/driver standpoint. After a lot of digging, I'm leaning toward a DIY RAID NAS running free BSD and ZFS as the file system.

In this configuration, as I understand it, you can build a headless server box with as many drives as you want and/or are necessary for your config, and ZFS runs the array as a software RAID. ZFS (https://en.wikipedia.org/wiki/ZFS) uses copy-on-write for data integrity, end-to-end checksumming, and is a self-healing file system; and that is only a few of the many(!) good things that it can do natively.It's hot swappable, guards against data corruption and is expandable to 16 Exabytes of storage (2^64). OTOH, I don't think it does windows...:~) It's also Open Source, runs faster without a hardware RAID card and is therefore more inexpensive. Sure, you'll pay for the Thunderbolt connects and 10GB/s--theoretical--throughput, but the savings from the Hardware RAID card can buy a Lot of RAM to make your server fly.

Since you want the Server in another room you can connect the Thunderbolt up to 30 mtrs, and I suppose you could daisy chain the whole enchilada if you wished.

As I understand it, there are relatively easy and straightforward ways to get ZFS to play with OSX and Windows.

For the record, my build uses the Gigabyte GA-Z77X-UP55TH MOBO, 32GB RAM, i7-3770K CPU and an EVGA GTX680 GPU. FCP 7 Studio on the MAC side and Adobe CS 6 on the Win 7 Ultimate side.

I'd like to hear anyone else's thoughts concerning this solution. Do you think it's as good as a hardware solution? What about Bang-for-the-buck comparing the two?

 

[BrodyJ]

Thanks everyone for you contribution,

I have been doing more research and reading and made some more calls.
The Areca 1882x seems indeed like a very good HW raid solution and is 100% compatible with a Mac (thanks Firewire2). It's very fast and apparently quite reliable. And from what I've gathered I can go well over 1000mb/s with more than 8 drives.

I'm quite happy now because I began this thread not knowing at all what I was gonna do and do now I have a pretty clear idea.
Now the last piece of the puzzle has to do with the Raid enclosure itself. I've been looking at the SansDigital products, the iStoragePro and others or simply the idea of recycling an old chassis and going DIY.

My main question about all of this is: does it make a difference? Could the Raid chassis be an issue or an eventual bottleneck or it's all about the controller and configuration? Could it have an influence in the speed, reliability, stability, performance, etc? I refer to that review of SansDigital bay review where the guy says the box never worked for him.

I'd be willing to go over my initial budget of $1000 to go the iStoragePro route if I gain something other than cosmetic or construction quality improvements.

What do you think?

 

[BrodyJ]

Concerning Snowroute's input, I was kind of considering ZFS SW raid solution as well but that seemed less plug and play and a less of a tried and tested route. (not the say it may be a very good option)
I had found this http://moderntoil.com/?p=509

What do you guys think?

 

[popatim]

While its certainly do-able and used often in the home server world, you would still need a high speed interconnect to the other room where you want to house it so you're looking at a few grand in 10G network stuff.

Alternately you could use an external connector version of that SAS adapter and house the drives with power supply in a pc chasis and keep it right next to your workstation, running the sata breakout cables in thru the back to the drives.

 

[Snowroute Towzone]

Concerning Snowroute's input, I was kind of considering ZFS SW raid solution as well but that seemed less plug and play and a less of a tried and tested route. (not the say it may be a very good option)
I had found this http://moderntoil.com/?p=509

What do you guys think?

Note: this response is conditioned on one of Brody's original parameters: "I'd like to do this under $1000, is that possible? (excluding the drives)...". As would I, including drives, or as much as is possible.

I also initially thought that the hardware RAID array solution was the way to go. I have been training on FCP 7 and was awaiting the new FCPX release; but as you may know, that seems to be a step back for many video editors. That said, there's a lot to like about the MAC OS. The breaking point for me came with the new MAC Pro's--nice(?) design, limited internal expandability; most anything expandable, by nece$$ity external anyway. So I found myself having to learn CS 6 (and the Mercury Playback Engine rocks) and I had this kicking dual boot. Since the video files are essentially agnostic, they only need to live "somewhere" that is accessible and preferably high speed--and safe. (ZFS = checksumming end-to-end and it's self-healing)

Given the constraints of MAC hardware expandability going forward and for software future-proofing coupled with my budget, when I began my research I noticed a lot of talk on the Adobe hardware site about Areca RAID controllers, which are known good for Win but the hardware guys weren't much into MAC; but they did discuss hardware failure. And when (not if) the controller fails, do you have an exact same spare controller with same firmware lurking around? At that point, what do you do? (And let's not discuss what happens if this is in a professional production environment). Given all these considerations: hardware, redundancy--file and hardware--, cost (Areca 1882X = $799 on Amazon) and throughput (4K vid anyone?) the only solution I've seen (so far) has been ZFS. (When a drive fails in ZFS, hot swap in a new one and keep on truckin')

One further note: I'm not married to any one solution. But the cost of fast, large, commercial storage is not achievable "at my pay grade". I'd be interested to hear about any solution that doesn't break the bank (< 2.5-3K) and that was solid. You can build a headless giganto box for around 1K and stuff it with 16, 32 0r 64 GB ECC RAM (OK..with 64 GB and ECC more like 1.5K) At least this way, I can DIY, and there's a strong community of BSD/ZFS users. BTW: up until 2008 Don Brady was Apple's lead engineer on ZFS development for MAC OSX--which is also based o0n BSD--so HFS talks to ZFS even easier than ZFS to NTFS--which isn't difficult at all.

So that's where I am now, with almost exactly the same constraints/parameter as Brody. If there's another rea$onable solution, I'd love to hear it. Thanks guys.

 

[FireWire2]

Like popatim pointed out
A let assume the ZFS gives you the speed that you need... How do you access it FAST. With 10Gb NIC, you need both end
There go 1000's of dollars. And it;s limited to 10Gb, where the actual speed max out @ 800MB/s if you are lucky.

With HW RAID 1882x, if you get OVER 1000MB/s up to 1600MB/s, if you configured right

ZFS is good but I do not see it in Video Edting just yet

 

[BrodyJ]

Thanks for the precisions Firewire2 and popatim.
I'm gonna go with the Areca 1882x.

Any inputs concerning my question about the cases? Do you think it's worth it to go higher end like the iStoragePro?
Could the enclosure become the cause of a raid malfunctioning like that dude who reviewed the Sansdigital seem to indicate?

 

[FireWire2]

Or this:
http://www.amazon.com/Eight-RAID-Enclosure-controller-support/dp/B00EQCHMYA

 

[BrodyJ]

Yeah. Or this...
But can the box make a difference in reliability or be create issues? Like the guy who says he could never make it work from the link in my previous post. Or perhaps it's the user...

 

[FireWire2]

BrodyJ

The card that bundle to Sansdigital is troublesome raid card.

But BOX can be problem too. Due to 6Gb/s bus, the back plane PCB traces becomes critical. I have see the 6Gb/s back plane acting up, switching back and forth 3Gb/ and 6Gb and causing big headache

http://www.amazon.com/Eight-RAID-Enclosure-controller-support/dp/B00EQCHMYA won't get you 700MB~800MB, but certainly will give you 450~600MB/sec.

Beside, It's AMAZON, if it does not work return it . That what i love about AMAZON

 

[BrodyJ]

Right. I'm just trying to understand how all of this work.
I still plan to use the Areca 1882x and build a mini SAS. Only shopping for the box now. So I wouldn't use the bundled raid card in sans digital.

So, if I understand correctly, 6gb/s back planes could be an issue, and I guess I have less chance of having those issues with a more expensive chassis like the iStorage Pro?

 

[FireWire2]

That I can not answer, no exprerience I stay from the high cost products, my clients very cost concious

 

[BrodyJ]

The iStoragePro iT8 without the SAS expander is priced similarly to the product you listed. Close to $700. This is still a high cost in my mind for something that is just a box and some electronic components you could get off the shelf for peanuts. But maybe this is where I'm wrong.

What about an off the shelf JBOD rack?

So I'm very cost conscious too, but I also want this to be rock solid.