Sign in with
Sign up | Sign in
Your question

RAID with large number of drives

Tags:
  • NAS / RAID
  • Performance
  • Storage
Last response: in Storage
Share
a b G Storage
May 26, 2011 9:55:02 PM

In another thread someone has talked about trying to get to a RAID 5 array to operate at 5GB/s with a substantial number of drives.

Thinking purely about redundancy, rather than performance, is it even possible to have a single RAID array with say 50 x 2TB drives and not be constantly replacing and rebuilding the array?

I'm discounting having a 50 drive RAID 1 array obviously because the capacity is so ridiculously low!

I'm not even sure how well the somewhat obscure RAID 15 would do with this volume of drives?

I'll point out this is all purely academic. I'm not planning on implementing such a system.

More about : raid large number drives

a b G Storage
June 29, 2011 1:49:38 PM

No one has any thoughts on this? :) 
m
0
l
June 29, 2011 2:12:11 PM

I see your point. At some point (and I don't know if there is an exact number) you will have so many drives that you may have to worry about the regularity of drive failure. Although that being said, you could have just as bad luck with an array of lets say 10 drives. Although sometime drive failure has a bit of warning, a lot of the time it's just a pure surprise. However, I think with a RAID as big as 50 drives you would have at least 2 or 3 hard-drives ready in backup just in case.

As for the 50 drive RAID 1 array - I think at some point with RAID 1 you are getting a bit carried away. Sure redundency is very important, but at what point is it just overkill? RAID 5 would be a bit different, and I've worked in a few engineering firms and this is what they use for their main servers. Right now my office servers have 20 TB free, and talking to the tech guy they don't use RAID 1 to achieve this.

A bit of sidetracking from your main points, but to answer your main questions:

- a 50 Drive array would be possible, but your right... with that many drives you may get to the point where you are replacing them often. This could be solved with keeping some spares in stock always, and the problem could be also as common in a smaller array.

- Obviously a 50 Drive RAID 1 array is extreme overkill IMO. I think the money could be better spent elsewhere (like on better backup) and then you wouldn't have to worry about losing data.
m
0
l
Related resources
a b G Storage
June 29, 2011 3:07:23 PM

A 50 disk raid 1 would be absolutely ridiculous for sure.

What does your office use to achieve the > 20TB capacity?
m
0
l
June 29, 2011 3:20:47 PM

I'd have to ask the IT guy personally, but I assume some sort of a RAID 5.

They recently went through an upgrade so I assume they are using 2TB drives since they are more of a proven size these days (3TB still a bit too expensive and untested). So I guess they are using 11 x 2TB drives to achieve the 20TB capacity.
m
0
l
a c 415 G Storage
June 29, 2011 11:46:58 PM

Rusting In Peace said:
Thinking purely about redundancy, rather than performance, is it even possible to have a single RAID array with say 50 x 2TB drives and not be constantly replacing and rebuilding the array?


Let's crunch some numbers. First of all, I'm going to assume that rebuilding will require the controller to read the remaining data from each drive sequentially. A smart controller will rebuild one stripe of data by issuing reads in parallel to all of the remaining good drives, but with 50 drives I think it's safer to assume that the controller may run out buffer space and that it would revert to sequentially rebuilding the data. At any rate this assumption gives us a "worst-case" scenario.

So that means to rebuild a 100TB array you'll need to read 100TB of data sequentially (you actually read 98TB and write 2TB but let's just say that we read 100TB). At 100MByte/sec it takes 1 million seconds to read 100TB - that works out to 11-1/2 days.

In terms of failure rates, the Google Study on Failure Trends in a Large Disk Drive Population seems to suggest an annual failure rate of around 5% - so for a 50-drive array that works out to 2-1/2 drives a year.

At 11-1/2 days per rebuild and 2-1/2 failures per year you could expect to spend almost month each year with your array in rebuild mode.

But the real problems are that:

a) Performance during the rebuild will be terrible, and if it's a busy volume then the rebuild will take a lot longer.
b) Chances of a second drive failure during a rebuild are awfully high.
m
0
l
a c 415 G Storage
June 29, 2011 11:56:56 PM

JordoR said:
However, I think with a RAID as big as 50 drives you would have at least 2 or 3 hard-drives ready in backup just in case.
Having spare drives eliminates the time to recognize the problem and initiate a rebuild - but it doesn't eliminate the rebuild time. It doesn't matter how many spares you have, they won't save you if you get a second drive failure during the rebuild. And the more drives you have, the longer the rebuild.

> Obviously a 50 Drive RAID 1 array is extreme overkill IMO.

Whether it's overkill or not depends on your requirements. RAID 1 (or 1+0) is the only way to get redundancy without pitiful write performance, and without having horrible performance degradation during rebuilds. There are a lot of companies with performance-critical line-of-business applications where RAID-1 is the only possible way to deliver the required service levels.

Where performance isn't as critical but huge volumes are required, RAID-6 is the appropriate solution. It virtually eliminates all of the rebuild risks associated with RAID-5.
m
0
l
a b G Storage
June 30, 2011 12:02:11 AM

Thanks for the example, clearly points out that this is unfeasible.

The remaining question is however, how do these data centers manage such large capacities?

For example, here's a Cisco case study for an Italian bank. They say they are using a IBM DS8100 Storage Array. Wouldn't such a set up suffer from the same issues? Even if the storage arrays are distributed across multiple machines the failure rate for drives are still the same?
m
0
l
a c 415 G Storage
June 30, 2011 12:59:02 AM

The fact that they're using an IBM DS8100 speaks volumes. This is a very high-capacity, enterprise-level storage product that can host over 1,000 disk drives. Businesses that deal with these kinds of products wouldn't hesitate to use RAID 1+0 for 60TB of storage. And if they needed the aggregate I/O performance they wouldn't hesitate to create that storage using 120 disks of 1TB each or even 240 disks of 500GB each rather than 60 disks of 2TB each.

In fact what these kinds of storage subsystems are designed to do is to take hundreds of drives running in RAID 1+0 and then carve up the space into logical volumes that are then allocated as needed to various servers. At this level of storage management you don't waste time worrying about the needs of individual servers, you create a back-end storage system that just plain works and then deal it out as necessary to the servers and applications that need it.
m
0
l
a b G Storage
June 30, 2011 1:08:24 AM

Take a look at Compellent's enterprise class solution, it let's you use RAID5/6 without the write penalty normally associated with parity RAID. It has a bank of high performance (even SSD) drives in RAID 10 that it uses for writing and these changes are migrated to the RAID 5/6 array over time. The contoller keep track of where each block of data is located so the effect is something that performs like RAID 10 but with the capacity of RAID5.
m
0
l
a b G Storage
June 30, 2011 9:15:21 AM

Sminlal your storage knowledge is frankly astounding. Is this part of your job or do you just read a lot?

So say I have a DS8100 and I decide to use 240 x 500GB in RAID10 giving a nice 60TB capacity. Making the massive assumption that 500GB have the same failure rate as 2TB drives, then I'd have approximately 12 drive failures per year. Given that a raid 10 system can fail when two drives in each side of the mirror fail, surely this is more risky than running fewer higher capacity drives? Would people actually take his hit for the increase in performance?

Thanks for your post tokencode. I guess that is a sort of tiered storage implementation. I like to make up storage terminology. It amuses me.
m
0
l
a c 415 G Storage
June 30, 2011 11:13:37 AM

I was a data centre administrator many years, so I have some experience with the kinds of requirements that companies can have. Uptime is almost always the highest priority, followed by performance and/or cost, depending on the application.

If your business requires a certain level of IOs/sec performance and the only way to achieve it is by using more, smaller drives then that's what you've gotta do. Ideally the application developers or the software vendor can optimize the program to reduce the hardware requirements, but in my experience by the time you find out there's a problem it's way too late in the game to make those kind of wholesale changes to the software - it ends up being cheaper to throw hardware at the problem than it is to spend the money and waste the time trying to retrofit the software.

Most storage subsystems can handle multiple failures in a RAID 1+0 set as long as exactly opposing drives don't both fail. So in a 240 drive array, if exactly the right drives failed, you could actually have 120 drive failures and still keep right on trucking without too severe an impact on performance. Of course you can't count on that many drives failing exactly the right way - still it's extremely unlikely that you're going to get enough simultaneous failures in exactly the WRONG combinations to cause a problem. And given that rebuilds on a RAID-1+0 array are a LOT faster than those of RAID-5, the likelihood of a second failure occurring during a rebuild is fairly low in the first place.

If you're running something that simply cannot fail (like a man-rated system where failure could result in loss of life), then you can always to go RAID1+0 systems that maintain three or more copies of the data.

Actually businesses that live or die on their data (such as banks) often lease communications lines fast enough to keep multiple mirrored copies of critical data in geographically separated locations - think of a RAID-1+0 set in New York mirrored to another RAID 1+0 set in Chicago. That way they're protected not only against hardware failure but also against almost all other risks such as utility / communications failures, natural disasters, etc. In scenarios like that the biggest risk by far is from somebody screwing up maintenance tasks such as program or system upgrades.
m
0
l
a c 78 G Storage
June 30, 2011 9:28:29 PM

Yes we built ONE 128x 2.0TB HDD raid 50 (256GB RAW) for DATA CENTER in Virgina, it wasn't bad at all
m
0
l
!