The TeraTower: 10.5 TB RAID, 12 TB JBOD

[scalar]

<b>Goal:</b> Maximum storage, RAID-5, minimum cost
<b>Hard drives:</b> forty SATA 300 gig
<b>Raw storage capacity (JBOD):</b> 12 TB
<b>RAID-5 parity consumes 13%:</b> 1.6 TB
<b>Total failure-protected capacity:</b> 10.4 TB

As an experiment, I'm trying to figure out what is the cheapest possible way to get an insane amount of storage for a home-based media server. RAID is important, since I don't want hardware failures destroying data. Here is one possibility, an 8 terabyte monster which would use NTFS dynamic disks to stripe multiple individual RAID-5 arrays into one single volume.

<A HREF="http://www.jpj.net/~scalar/teratower.gif" target="_new"><b>26-bay tower, with eight 5-drive bays (mockup image)</b></A>

$550, Qty 1, 26-bay (all 5.25") Server case
$428, Qty 1, SuperMicro X6DHE-XB E7520 motherboard, 6 PCI-X slots
$211, Qty 5, Highpoint PCI-X SATA-RAID controller, 8 drives
$113, Qty 8, Supermicro 5-bay SATA enclosure, uses three 5.25" bays
$185, Qty 40, Maxtor 300gig 7200rpm SATA hard drives
$ 89, Qty 2, Enermax 460 watts Dual Fans 24+8+4pin, 8big+2small
$269, Qty 1, Intel Xeon 2.8 GHz with EMT 64, 800MHz FSB, 1MB L2 Cache
$ 89, Qty 2, Crucial 184 Pin 512MB ECC DDR PC-3200, OEM
$ 22, Qty 1, DVD drive
$ 10, Qty 1, floppy drive
$ 5, Qty 1, 5.25" floppy mount

Cost per raw gig: $10999 for 12 TB, $0.89 per gig
Cost per RAID gig: $10999 for 10.4 TB, $1.05 per gig


I'm not really liking this plan too much due to the high costs of the PCI-X motherboard and the need for separate power supplies. With all the specialty components, it would need about $1500 worth of hardware just for the case, motherboard, power supplies, CPU, and memory, before I even get the first terabyte installed.

I'm looking at making smaller servers with just 8 drives per system, but AFAIK you can't merge RAID volumes across multiple servers with NTFS, so that limits the unified-storage capacity..


<b>Edit:</b>

1. lesser power supplies are okay, no need for 2 700w
- estimated power consumption per drive: 10 watts
- 10w * 40 = 400 watts
- supermicro recommends 420w for motherboard

2. added CPU, memory, CD/DVD, and floppy costs to flesh out costs (newegg)

<b>Edit #2</b>

3. I had done some incorrect RAID5 efficiency calculations. A bigger array means more space available in the array set. An 8-drive array allows 87% efficiency rather than merely 66%, and gives a much lower cost per gig..<P ID="edit"><FONT SIZE=-1><EM>Edited by Scalar on 03/20/05 08:06 PM.</EM></FONT></P>

 

[folken]

I don't think NTFS is goona work on there if it is made into one 8TB volume. NTFS can only support somewhere around 2TB I believe. You will need to put a linux distro or unix on there. That would make it more secure/reliable anyway, lol.

<A HREF="http://www.folken.net/myrig.htm" target="_new">My precious...</A>

 

[scalar]

2 TB is the limit for a bootable NTFS partition. That's a limit imposed by the drive's partition table. But you can have dynamic NTFS disks which are far larger...

<A HREF="http://www.microsoft.com/resources/documentation/Windows/XP/all/reskit/en-us/Default.asp?url=/resources/documentation/Windows/XP/all/reskit/en-us/prkc_fil_tdrn.asp" target="_new"><b>Microsoft KB: Maximum Sizes on NTFS Volumes</b></A>

<i>In theory, the maximum NTFS volume size is 264 clusters minus 1 cluster.
However, the maximum NTFS volume size as implemented in Windows XP Professional
is 232 clusters minus 1 cluster. For example, using 64-KB clusters, the maximum
NTFS volume size is 256 terabytes minus 64 KB. Using the default cluster size
of 4 KB, the maximum NTFS volume size is 16 terabytes minus 4 KB.

Because partition tables on master boot record (MBR) disks only support
partition sizes up to 2 terabytes, you must use dynamic volumes to create NTFS
volumes over 2 terabytes. Windows XP Professional manages dynamic volumes in a
special database instead of in the partition table, so dynamic volumes are not
subject to the 2-terabyte physical limit imposed by the partition table.
Therefore, dynamic NTFS volumes can be as large as the maximum volume size
supported by NTFS. Itanium-based computers that use GUID partition table (GPT)
disks also support NTFS volumes larger than 2 terabytes.</i>
[...]


I don't think I'd want to boot off the thing anyway. Better to use a separate drive, so if Windows crashes and burns you only rebuild one drive..

 

[scalar]

Note, I changed the first post, because I found I've miscalculated RAID efficiency. You waste far less space as the array size increases, up to 87% usable space for an 8-drive array:

8 drive array has 1 parity volume
Number of data drives = 7
efficiency = n/(n+1)
Efficiency is 7/8, or 87%

So, the total space is likely <b>10.4</b> gigs rather than merely <b>8</b> gigs. Woohoo!



The way to make an insanely huge unified RAID volume:

a. Five individual 8-drive RAID arrays: 2.088 TB per array
b. Windows NTFS dynamic RAID-striping: Stripe across all five arrays
c. 2.088 * 5 arrays = one 10440 gig NTFS volume

RAID0-on-RAID5 protection: <b>5 parity drives</b> protect entire <b>40 drive</b> stripe array

<P ID="edit"><FONT SIZE=-1><EM>Edited by Scalar on 03/20/05 08:20 PM.</EM></FONT></P>

 

[z2895]

It sounds like it might be a personal media server? If so then you probably don't need good access time or throughput?

What about simply buying the cheapest $/GB drives on the market and buying USB converters for them all, I'm pretty sure you could then hook all drives up to only ONE computer with by filling it with USB controller cards. You can then also create your own powersupply for the drives by getting a used high quality 5V+12V industrial power supply. It should be very easy to create and this it has to be the cheapest possible method of creating a big array out of standard components.

Also the easiness of setup and expandibility of this system would be unsurpassed, the only drawback of it is obviously the speed of the array being limited by USB 2.0, but as a media server this may not be an issue for you. I think you could even get around the speed issue by configuring the drives in the same RAID array on different USB controllers if you know what I mean.