Hi all -
Starting my research for a new project: Replacing my current 3.5TB NAS home server. The data drives in that server are over 5 years old, with 1 of the 7 having already failed. Its Raid 6, so I'm still good, but they were all bought at the same time, so I'm likely on borrowed time.
So I got started looking around, and quickly become overwhelmed. Rather hope this excellent community can help.
First the requirements:
I need to replace my 3.5TB of usable storage. This is something I would like to do no more than once every 5 years or so. I love using Linux MDADM for a variety of reasons (like being able to upgrade a 6 member raid set to 7, reshaping it on the fly, after replacing a 250gb boot drive with a 1tb one and using the extra 750gb partition as the 7th "drive").
Not concerned about a controller failure - the raid set would just go offline until I replaced the mobo or SATA card. Its a home NAS, stores movies, photos, etc. Nothing I can't live without for a week.
Drive speed isn't an issue, the box is hardwired to my home 1gbit Ethernet - any modern drive will be fast enough for streaming that network to death, and when stripped across a raid group, even more so. Likewise this doesn't take much of a CPU. The current dual processor typically runs at most a few percent busy (unless I have BOINC cranked up of course! (which I normally due, because I hate wasting CPU cycles)).
Of course, the server is full. If I've going to rebuild, I should buck up for higher capacity drives.
Since all the drives are old, as is the motherboard and CPU, I'm thinking its best to just start over, build a new server, and copy the data over. That way I'm not man-handling drives, replacing the wrong one, etc. I'm not having any problems with the CPU, but 5+years is a pretty good run, I should give the guy a break and not expect another 5 from him.
General thoughts/concerns to date:
Cost is a concern - duh...
"Consumer disk", like the WD greens and blues, have 2 year warranties. I'm guessing that reflects their estimated MTBF and read-error-rate stats. Probably not a good fit, since I'm guessing getting 5 years from them is unlikely.
"TLER" is meaningless for me - MDADM doesn't care, I don't care, if the drive hangs for 30 seconds its no big deal. Of course, if they all do that once a minute, I care, but something is seriously wrong if that starts happening.
"Read Error Rates", typically seen as 1 in 10^13, is starting to be as concern to me with modern multi-terabyte drives in raid. I've read that even with Raid 6, loss of 2 drives might prevent a rebuild due to a read error while reconstructing. This kind of makes sense to me, although if all I needed to do was restart the rebuild, I wouldn't care. I've seen a review or two that some Enterprise drives are now being rated at 1 in 10^15, or 1/100th the read error rate, which feels significant.
I'm finding it rather hard to get good, real, numbers on the different drives. Things like MTBF, read-error-rate, etc. Sometimes even basics like RPM are not being displayed by some drive manufactures, let I'll find a random web review that has the data. Unfortunately, what I find while reading my iPhone while waiting somewhere is often lost by the time I come home and want to find it again.
Need to find a balance. Suspect 8 drives in a full size tower is probably about the limit, given the desire to have a CDRom as well for installation purposes. That is a guess on my part, but I presume I can find a reasonably cheap tower that can handle that. I don't care what it looks like - it will be sitting in my server room (e.g. my computer corner in the basement). Anyhow, since I'll be using Raid 6, more drives is better, with (8) drives providing me the capacity of (6) (e.g. 75%) If I used 5 drives, I'd only get 60%, and 4 would give me 50%, but you all know that already. Alas, doubling the capacity of a drive seldom doubles its cost, but there are financial limits... (Having dreams of (8) WD RE drives, and nightmares at the cost!).
Need to find a balance part 2: After fooling with PCs since, well, before there were PCs (I had a really zippy Z-80 with 48K of memory that could compile Fortran and C programs, freaks me that MS Windows needs gigabytes...), I know that the latest and greatest often costs the most. Generally there is a "knee in the curve". Doing a quick search, using WD AV drives (no TLER, good enough speed, 3 year warranty) as an example:
500GB - $63 = $126/TB
1TB - $80 = $80/TB
2TB - $100 = $50/TB
3TB - $140 = $46.67/TB
4TB - $190 = $47.50/TB
Hmmm... pretty linear between 2TB, 3TB, and 4TB... surprised, but cool! Clearly (8) 2TBs with 12TB usable would be a better buy than (4) 4TBs with "only" 8TB usable. Although... If I started with (4) I could grow to (8) at a later time, presumably for less money, and just reshape the Raid group (and resize the underlying filesystem, but that just takes seconds). Kind of like that second option, both for the flexibility, and to allow me to get different generations of drives (helping to spread out the lifetimes between the drives so they don't all start failing at once like I currently am concerned about).
Finally, what about AV drives? Cool... low power... good enough speed... designed for 24x7 operations in hot servers...mid-range warranties? Is there something NOT to like for this application?
From a cost perspective, I'd pay extra for drives that would last longer, but there are limits (e.g. I wouldn't pay double to get 5 year drives instead of 3 year ones).
I've considered buying a semi-cheap 4TB external USB drive (~$180) to back everything up and then start swapping disk and continuing to use my current case/mobo/cpu. That would probably save me some cash for now and I could throw the USB drive onto my backup server after this project is complete. However, since the ones I've looked at use WD Green drives, I'd be nervous about that failing fairly soon (reviews have not been great on those units in terms of lifetimes). Hmmm, maybe buy an external USB case for $10, throw a 4TB disk in it, then after everything is moved, add that drive into the array and reshape... anyhow, if I opt to reuse the existing system, something will need to hold a backup of my data or I'll freak while doing the shuffle.
So my questions:
1) Does Tom or someone else have a consolidate list of basic drive stats somewhere? Looking for drive class, MTBF, and read-error-rate as basics.
2) What am I missing in my considerations above?
3) Is there a clear winner for this project? (Of course, if so, WHY???)
4) Are there clear losers for this project? (Like perhaps AV drives because I'm missing something basic?)
5) Beyond the disk, any motherboard/cpu recommendations? This is a server, I use Gentoo Linux at the command line, so something with low-end video will be fine (most of the time I SSH in anyhow). #1 priority for the motherboard/cpu is reliability. #2 is built-in SATA ports. #3 is cost. 3gb SATA ports are fine, again speed is not an issue for this project - I don't care if a reshape takes a day vs. a half-day.
6) Given no fancy video card (perhaps one heat-sink instead of fan based for reliability if the mobo doesn't have video built in), how big of a power-supply do I need. Suspect I have a good brand 650W around here as a spare...
7) Case recommendations? Drive cooling is the priority here, with (8) internal bays riggable one way or another.
8) Any easy way to relate a physical drive to the Linux SD number? Building this system will be easy relative to figuring out which of the 4 to 8 drives has failed sometime in the future. That is one reason I'm looking to replace, vs. rebuild, my current environment.
If you have made it to here... THANK YOU.
Cassey
Starting my research for a new project: Replacing my current 3.5TB NAS home server. The data drives in that server are over 5 years old, with 1 of the 7 having already failed. Its Raid 6, so I'm still good, but they were all bought at the same time, so I'm likely on borrowed time.
So I got started looking around, and quickly become overwhelmed. Rather hope this excellent community can help.
First the requirements:
I need to replace my 3.5TB of usable storage. This is something I would like to do no more than once every 5 years or so. I love using Linux MDADM for a variety of reasons (like being able to upgrade a 6 member raid set to 7, reshaping it on the fly, after replacing a 250gb boot drive with a 1tb one and using the extra 750gb partition as the 7th "drive").
Not concerned about a controller failure - the raid set would just go offline until I replaced the mobo or SATA card. Its a home NAS, stores movies, photos, etc. Nothing I can't live without for a week.
Drive speed isn't an issue, the box is hardwired to my home 1gbit Ethernet - any modern drive will be fast enough for streaming that network to death, and when stripped across a raid group, even more so. Likewise this doesn't take much of a CPU. The current dual processor typically runs at most a few percent busy (unless I have BOINC cranked up of course! (which I normally due, because I hate wasting CPU cycles)).
Of course, the server is full. If I've going to rebuild, I should buck up for higher capacity drives.
Since all the drives are old, as is the motherboard and CPU, I'm thinking its best to just start over, build a new server, and copy the data over. That way I'm not man-handling drives, replacing the wrong one, etc. I'm not having any problems with the CPU, but 5+years is a pretty good run, I should give the guy a break and not expect another 5 from him.
General thoughts/concerns to date:
Cost is a concern - duh...
"Consumer disk", like the WD greens and blues, have 2 year warranties. I'm guessing that reflects their estimated MTBF and read-error-rate stats. Probably not a good fit, since I'm guessing getting 5 years from them is unlikely.
"TLER" is meaningless for me - MDADM doesn't care, I don't care, if the drive hangs for 30 seconds its no big deal. Of course, if they all do that once a minute, I care, but something is seriously wrong if that starts happening.
"Read Error Rates", typically seen as 1 in 10^13, is starting to be as concern to me with modern multi-terabyte drives in raid. I've read that even with Raid 6, loss of 2 drives might prevent a rebuild due to a read error while reconstructing. This kind of makes sense to me, although if all I needed to do was restart the rebuild, I wouldn't care. I've seen a review or two that some Enterprise drives are now being rated at 1 in 10^15, or 1/100th the read error rate, which feels significant.
I'm finding it rather hard to get good, real, numbers on the different drives. Things like MTBF, read-error-rate, etc. Sometimes even basics like RPM are not being displayed by some drive manufactures, let I'll find a random web review that has the data. Unfortunately, what I find while reading my iPhone while waiting somewhere is often lost by the time I come home and want to find it again.
Need to find a balance. Suspect 8 drives in a full size tower is probably about the limit, given the desire to have a CDRom as well for installation purposes. That is a guess on my part, but I presume I can find a reasonably cheap tower that can handle that. I don't care what it looks like - it will be sitting in my server room (e.g. my computer corner in the basement). Anyhow, since I'll be using Raid 6, more drives is better, with (8) drives providing me the capacity of (6) (e.g. 75%) If I used 5 drives, I'd only get 60%, and 4 would give me 50%, but you all know that already. Alas, doubling the capacity of a drive seldom doubles its cost, but there are financial limits... (Having dreams of (8) WD RE drives, and nightmares at the cost!).
Need to find a balance part 2: After fooling with PCs since, well, before there were PCs (I had a really zippy Z-80 with 48K of memory that could compile Fortran and C programs, freaks me that MS Windows needs gigabytes...), I know that the latest and greatest often costs the most. Generally there is a "knee in the curve". Doing a quick search, using WD AV drives (no TLER, good enough speed, 3 year warranty) as an example:
500GB - $63 = $126/TB
1TB - $80 = $80/TB
2TB - $100 = $50/TB
3TB - $140 = $46.67/TB
4TB - $190 = $47.50/TB
Hmmm... pretty linear between 2TB, 3TB, and 4TB... surprised, but cool! Clearly (8) 2TBs with 12TB usable would be a better buy than (4) 4TBs with "only" 8TB usable. Although... If I started with (4) I could grow to (8) at a later time, presumably for less money, and just reshape the Raid group (and resize the underlying filesystem, but that just takes seconds). Kind of like that second option, both for the flexibility, and to allow me to get different generations of drives (helping to spread out the lifetimes between the drives so they don't all start failing at once like I currently am concerned about).
Finally, what about AV drives? Cool... low power... good enough speed... designed for 24x7 operations in hot servers...mid-range warranties? Is there something NOT to like for this application?
From a cost perspective, I'd pay extra for drives that would last longer, but there are limits (e.g. I wouldn't pay double to get 5 year drives instead of 3 year ones).
I've considered buying a semi-cheap 4TB external USB drive (~$180) to back everything up and then start swapping disk and continuing to use my current case/mobo/cpu. That would probably save me some cash for now and I could throw the USB drive onto my backup server after this project is complete. However, since the ones I've looked at use WD Green drives, I'd be nervous about that failing fairly soon (reviews have not been great on those units in terms of lifetimes). Hmmm, maybe buy an external USB case for $10, throw a 4TB disk in it, then after everything is moved, add that drive into the array and reshape... anyhow, if I opt to reuse the existing system, something will need to hold a backup of my data or I'll freak while doing the shuffle.
So my questions:
1) Does Tom or someone else have a consolidate list of basic drive stats somewhere? Looking for drive class, MTBF, and read-error-rate as basics.
2) What am I missing in my considerations above?
3) Is there a clear winner for this project? (Of course, if so, WHY???)
4) Are there clear losers for this project? (Like perhaps AV drives because I'm missing something basic?)
5) Beyond the disk, any motherboard/cpu recommendations? This is a server, I use Gentoo Linux at the command line, so something with low-end video will be fine (most of the time I SSH in anyhow). #1 priority for the motherboard/cpu is reliability. #2 is built-in SATA ports. #3 is cost. 3gb SATA ports are fine, again speed is not an issue for this project - I don't care if a reshape takes a day vs. a half-day.
6) Given no fancy video card (perhaps one heat-sink instead of fan based for reliability if the mobo doesn't have video built in), how big of a power-supply do I need. Suspect I have a good brand 650W around here as a spare...
7) Case recommendations? Drive cooling is the priority here, with (8) internal bays riggable one way or another.
8) Any easy way to relate a physical drive to the Linux SD number? Building this system will be easy relative to figuring out which of the 4 to 8 drives has failed sometime in the future. That is one reason I'm looking to replace, vs. rebuild, my current environment.
If you have made it to here... THANK YOU.
Cassey
Facebook
Twitter
Instagram