binbon1009

Distinguished
May 29, 2009
9
0
18,510
Hi everyone,

My boss wants me to get a 96TB RAW storage NAS. Just wondering how could it be done and how much will it cost? Need to integrated with our existing 8g fiber switch. Please help, I need to get this done in 2 weeks. I did some research but couldn't find any solution for complete this project. Need all 96TB in one server unit.
 

sub mesa

Distinguished
Assuming you want to use consumer 1-2TB disks, such as the WD Green 2TB disks, i would get this picture:

96/2 = 48 disks required, or two 24-port Hardware controllers.

I'm a software RAID guy, but with so many disks you NEED staggered spin-up, so two 24-port Areca hardware RAID adapters using PCIe x8 interface would be logical. This could be done by a normal (quadcore?) system with a normal SLI motherboard. Such a system would go for:

CPU: 225,-
Mem: 75,-
Mobo: 100,-
PSU: 100+
casing: 200,-
Areca ARC-1280ML (24-port) adapter: 1000,- x 2 = 2000,-
48x 2TB disks = 48 x 225 = 10.800,-

Total: 13.500,- (euro)

Ofcourse this is raw storage space, and not taking the difference between MB/s (1000) and MiB/s (1024) in account. 96TB would display as 87.3TiB raw space, if you use (multiple) redundancy layers like a spanned RAID6 array it will reduce the usable space even further. So think about what usable space you require.

Also, for such systems i would recommend testing it thoroughly and usage of modern filesystems may be desireable to keep the system operating reliable and fast. If you intend to run this on Windows you should do research into any limitations you may encounter. Also, NTFS may not be the best filesystem for such a serious setup, but that's for you to judge.
 

sub mesa

Distinguished
Considering he said "my boss" i assume this is business-related. But i don't hope they want to trust critical data on a system that is not properly tested yet before it is taken into production. That is potentially very dangerous.
 

jrst

Distinguished
May 8, 2009
83
0
18,630
A little more info, such as how you're going to use it, your budget, whether you're expecting to do this as a DIY, and whether you expect someone to support it would help refine the solution. Is this mission-critical? Does it need to be operational in 2 weeks, or just have it designed and the parts ordered? Are your primary protocols CIFS/SMB, NFS, iSCSI, FC/SCSI, or what?.

sub mesa has pretty much covered the DIY approach, although you'll also need at least one 8G FC adapter, which will add ~US$2500 to the price. However, note that DIY config doesn't provide controller/interface redundancy, or expansion beyond 48 disks / 96TB unless you go dual-port, SAS expanders, and more robust frames/cages, all of which will add significantly to the price. (Not to mention adding weeks or months integrating the parts and ensuring they work together.)

That said, I assume (but obviously may be wrong), that anyone looking for 96TB of 8G FC-connected storage isn't playing in the minor leagues, and isn't in the market for a DIY solution (and if they are, they probably need a whack with a very big clue stick). 8G FC? Really? Someone has expensive tastes, or has serious needs, and I assume that isn't just for grins-and-giggles, because the fabric to support it ain't cheap. Building such systems is Serious Business, requires Serious Engineering, and comes with Serious Price Tags.

I suggest picking up the phone ASAP and calling: Sun, Dell, HP and EMC. Of those, IMHO, the Sun 7310 (~US$40K starting price) and the 7410 (~US$60K starting price) offer the best bang-for-the-buck. (But expect to pay much more by the time you get done, especially for a 96TB system.) If your boss goes into sticker shock, offer the DIY solution, but only after very carefully CYA--and a lot more time and engineering work on your part--because it isn't nearly as easy as it might first look (integration of those oh-so-attractively-cheap DIY parts can be, and more often than not is, a gold-plated b*tch),. Not to mention that you're likely to be number-one on the receiving end of a massive sh*t-storm when it hiccups and all those systems that depend on it come crashing down.
 

cletus_slackjawd

Distinguished
Dec 26, 2006
347
0
18,790
Do you work at WESCO? You sound like my IT guy. I wouldn't do the DIY approach if it's your a$$ on the line. I'm curious if all the space is for engineering data with diagrams etc or a super large database.
 

alexmtu

Distinguished
May 29, 2009
2
0
18,510
There is a simple way to do this. Get a 8U 48 drives plus 2TB hard drive in it. 48*2=96TB, and you need 2x 24 port controller card (Areca 1281ML will do the job).
 

binbon1009

Distinguished
May 29, 2009
9
0
18,510

We do have TONS of files to backup. This is going to replace our current storage server which used to be our main storage server (going to be our backup server after we got this 96TB server)
 

binbon1009

Distinguished
May 29, 2009
9
0
18,510

I agree with you that something needs to hold our critical data need to be tested before we put it to work. I have 11 more guys on our team to do the configuration and testing.
 

binbon1009

Distinguished
May 29, 2009
9
0
18,510

8U? You mean a SINGLE 8U rack mount server chassis? Wow! Where can you get it? Does it have 48 hot-swappable drive bays? What kind of power supply does it come with? What kind of connection for the hot-swappable back-plan to the raid controller card? mini SAS? or direct connect?
Sorry for asking so much questions, but a single 8U server chassis is just what we are looking for. Please reply.
 

ralphael

Distinguished
May 25, 2009
9
0
18,510
my 2 cents....

I would suggest that you get more then 96TB for mirroring.

I would not trust 48 x 2TB drives on one raid...

the probability of 2 or more drives failing simultaneously is far greater then having just a few drives.

I would cluster them into groups of 4 or 8 raid 5/raid 6, then group them into raid 10 (mirror and striping) or any other raid that takes your fancy.

96TB is serious business and you will need a nice air conditioned room.

http://www.raidinc.com/falcon_III_48bay.php
http://www.pacstorage.com/index.php?page=816702.txt
 

sub mesa

Distinguished
You should address the points made by jrst, as he gave you good advice IMO.
Also, you should give us more information about your situation. Which operating system are you going to run? Etc.
 

alexmtu

Distinguished
May 29, 2009
2
0
18,510
I recommend to find any custom server manufacture to complete this job. Our company got pretty good quote from a local manufacture - RackMountPro. They custom build any type of configuration within 3 days. Since this is a unique server project, so I recommend you to contact them at 800-526-8650 for Michael.
www.rackmountpro.com
 

binbon1009

Distinguished
May 29, 2009
9
0
18,510

Thanks! Their 8U Chassis is great. I am waiting for the price quote from Dell, hp atm. This rackmountpro company submitted thier quote super fast (within 30 mins) The price is good compare to IBM. Never heard of this company but the guy you referring Michael, is nicely taking care all my questions. Will update the final spec and quot to THG once we decided which vendor to go for.
 

binbon1009

Distinguished
May 29, 2009
9
0
18,510

The www.pacstoreage.com their 4U 48 Bay will cause heat and vibration problem by the way how they place the drives.
 


Start by calling IBM, Dell, or some other hardware builder and start getting price quotes. Buying a pre-built system from a hardware builder also comes with a warranty, tech support, and a service plan.

I would not recommend a self built 96TB storage solution to house business critical data for any business, no matter how small or large. Neither would I recommend that you assume responsibility for maintaining a 96TB NAS if you need to come to these forums to search for an answer.

Don't mean to sound harsh, but if I learned that my IT guy was housing critical data on a self built storage solution that did not have a warranty or service plan, I would kick him in the arse and walk him out the door.

Seriously, call a professional.
 

jrst

Distinguished
May 8, 2009
83
0
18,630


The raidinc/pacstore (Xyratex OEM) approach is how very high density solutions are built these days. DirectData and Terascala are similar. DirectData puts 60 drives in 4U.

They work (at least DirectData's). They are bought by Really Big Organizations that spend Lots of Money and do Very Serious Work. Their engineers have a very good understanding of power, heat and vibration, how to manage it, and not piss off Very Big customers.

However, I wouldn't suggest them in your case because those densities carry a premium, especially if you don't need/use it, and are for people building near-line petabyte+ storage farms.

Your needs are better and cheaper met using more conventional lower density solutions. (Unless your rack space is severly constrained and carries an extreme premium. Even then I'd question the economics for a measly 96TB.)

To echo chunkymonster, if you're that far down in the the minutia evaluating solutions with limited knowledge, flawed assumptions, or based on what you read in these threads, you need to step back and call a professional.
 

Gatorbait

Distinguished
Jan 29, 2009
63
0
18,640




Just Go to the LSI web site: www.lsi.com
They do everything storage.
 

binbon1009

Distinguished
May 29, 2009
9
0
18,510
Thanks for all the recommendations. We order 2x 96TB servers from Rackmountpro.com (was going to order just 1 but they found us a better solution). Like I said, we need this up and running in 2 weeks but these guys at Rackmountpro.com made it happened in just 8 business days. They even did a 48 hours burn-in test before they ship these to monsters out. Besides the performance, the price is one of our concern too. With less than 30k (1.8k less from other vendors) for each system they provided free customized setup just for our company, an full system report and super fast customer service. I will definitely recommend these guys to everyone. All of their sales people have technical background, so I was pleased when I talked to them (they know everything they are selling).
*Special thanks to Michael (a sales person in Rackmountpro.com) for everything you've done helping us to complete this project in timely manner!