15 Years Of Hard Drive History: Capacities Outran Performance

pschmid

Distinguished
Dec 7, 2005
333
0
18,780
We've got quad-core processors, gigabytes of RAM and awesome graphics. But what has been going on with hard drives during all the time that these goodies have been developed? Capacities have multiplied and will hit one terabyte per drive soon, but as we'll see, performance hasn't been able to keep up the pace.
 
Hard Disk drives can only get so fast. I'm pretty sure that the top of the line SCSI's out there have topped out the RPM that you can spin the platters and still be able to read/write effectively.
 

icepop456

Distinguished
Nov 27, 2006
19
0
18,510
Can the editors proof read the article and change all the GB to MB for the 40MB drive? There are two obvious mistake (one being the headline introduction for the drive).

Recently Tom's hardware articles seem like they are rushed online...
 

funnyvlad

Distinguished
Jan 7, 2006
96
0
18,630
I agree, totally annoying. When I first read the article I actually thought it was a 40Gb drive that propably was used by NASA and costed about 20000$ or something like that. The difference between Gb and Mb is tousandfold so when I read something about it I shouldn´t think what it is.
 

shushikiary

Distinguished
Jul 31, 2006
24
0
18,510
I was an intern for maxtor about a year ago, and was going to work for them before they anounced being bought by seagate and then my job there dried up (right when I was graduating with my BSEE).


I can tell you there is a reason for the change in performance since the old hard drives. It's all due to density. You'd think that the more bits/second you pass by the head the faster you could read right? So higher density would give you better performance... but here's the issue... they dont write 1's and 0's to hard drives anymore. They USED to write 1's and 0's, using an up or down magnetically oriented field and then using peak detection (current caused in the head by the magnetic field is a negative deriviative, so when it just passes by the head you get a spike in voltage as the field on the disk is DC and the only AC part of the effect is beacuse it's moving past the head). Now they use what is called PRML and are switching to another form of a similar thing, it's rather complicated, so let me start explaining......

PRML: This is a genious idea by HD guys to read data off a disk that is smashed too close together, but it causes lots of performance hits. When density started increasing on hard drives, they had issues with the fields of adjacent bits interfiering with eachother, so if you wrote 111 you'd be fine, but if you wrote 101 you wont get a nice separated +1 spike -1 spike +1 spike, you might get +.5 spike -.3 spike +.5 spike (spikes are no longer large enough to register under peak detection). So they said... hey we have faster and faster channel chips (main big chip on the hard drive that controlls the reading/writing for the most part). So... why not over sample the signal? Like refreshing your computer screen, but because of the mathmatical principle called nyquist rate, you have to sample AT LEAST 2x the frequency that the bits are bassing under the head. So you over sample the signal, and to lock the sampling in you use a phase lock loop at the start of each sector..... then you over sample the data, and it looks like the 1's and 0's are interfearing... now here's the part that takes ALOT of processing power... You then go through what is called a Vitirbi (sp?) tree. It's a logic tree that changes what branch it branches to based on the MSE (mean squared error, a probability based number that tells you your error of the read value). This logic tree looks at the interference of the 1's and 0's (which have been written to the hard drive messed up because that's the only way you can do it). Then it looks at how much "error" is in the interference VS a randomly generated bit patter, then goes through the tree until the MSE is as small as possible, and hands over that value as the "data". AKA it guessed at your data using a fairly mathmatically sound method... works most of the time, cuz your hard drive works right? Yep... but that takes alot of processing power and time... and lots of floating point or Q based calculations...

the 620 gig drive maxtor should have out by now had a data frequency of 1 gigabyte/second at the outside of the disk, and the channel ran at 2 gigahurtz (to meet nyquist rate) (dont worry 0 point sampling is prevented by the PLL). The channel almost had more transistors in it than the processors in computers at that point in time (when I was working on it).

The next problem is this: When your data is packed so close together and your data's magnetic field is so small, it's easy to accidentally effect it, the higher the density, the higher this problem is. When you read an adjacent track on the hard drive, or especially write, you can damage the tracks next to it, so before a write to a track you have to read the adjacent tracks, then do your write, then re-write the adjacent tracks as well, and the higher your density the more you have to do this... causing more performance issues. (usually the read degradation is small and they just ignor it). There is ECC (error correcting code) in each sector that can correct a large data vault (I cant really tell you this number as it's a trade secret for each company, but I can tell you that it's upwards of half the bits in the sector).

Some of the hard drives have changed from using PRML to a new "changed" version of PRML, it's different in that instead of using a vitirbi tree it uses a node based forward and back propagation method, with propagation to nodes determined by a modified MSE, I cant give out too much info on it... but it's supposed to be more accurate, and at times can take less time to calculate, but it can also take longer, depends on how much effort you want on trying to get your "scambled" data out. Not to mention that the data is actually scambled when it's writtin on the disk, the data is multiplied by a psuedo random value (mathmatically chosen to prevent large number of adjacent 1's, or 0's, like 111111111, etc) and then unscrambled when read.


The good news: New recording methods can improve performance, bit density (hd size) and reduce processing overhead.... welcome to perpandicular recording! This technology has been worked on for years, and seagate finaly (notice AFTER they merged with maxtor they released this)... hehehe... I'm not bias, I swear... anyways... Normally data is written in a horizontal fashion on the disk, HOWEVER this new method alows the data to be written in a virtical fashion, so the field is oriented virtically through the disk. The center of the disk is a material that allows magnetic field flux through it, so you can still write to both sides of the disk... Because of the way the field fluxes work out, you no longer have as much bit interference, and you can back more data together... and get this.... When you write a 1 you might write (+1 for say field pointing up and -1 for field pointing down) +1+1+1+1+1, and for 0 -1-1-1-1-1... when each time the head passes over the series of field chunks, you get a spike... and if you filter all those spikes together... you get a nice smooth signal... welcome back to 1's and 0's!! So with perpandicular recording you get increased disk size, back to writting 1's and 0's and better data integrity, with increased performance due to less overhead (no PRML). YAY!

You guys should have tested one of seagates perpandicular recording drives in the line up..... i'd like to see what it will do, and look forward to looking forward to what they can do in the future.

Never think that technology has reached it's end... humans always somehow come up with a new way of getting what they want done... for what ever reason they want it.
 

galatine

Distinguished
Nov 27, 2006
1
0
18,510
It would also be interesting to have some information here about trends in the durability of harddrives over the years; newer drives seem to have far shorter lifespans than 'old' ones - my old Quantum fireball 6.2 is still going strong but I've burnt through endless modern Maxtor drives of far greater capacity over the same period. I suppose it's hard to determine whether this is the result of the higher stresses on the newer drives imposed by their higher performance or just that there's more money to be made by selling high-capacity drives that fail after a couple of years but are cheaply-replaced...
 

tleavit

Distinguished
Jun 28, 2006
145
0
18,680
I’ve been telling people this for years.

Everyone talks about CPU and GPU power and the latest high speed ram. No one ever talks about hard drives. So I'll say it again.

Instead of going out and buying that 500 gig western Digital... Go buy 2 250 gig drives and raid stripe them.

I now have 3 sets of of Western Digital raptor drives:

2 X 36
2 X 72
2 X 150 (sata2)

I have 7 systems at home and no matter what I put in a box, running 2 drives in stripping has by far the greatest impact to perf.


PS. I remember buying that Maxtor 40 meg hard drive for my old 386 16 and it was a beast!!!!
 

ninjahedge

Distinguished
Aug 3, 2006
3,210
0
20,780
A couple of things need to be taken into account.

First of all, the rate of data transfer depends on the mechanism for data transferral. I am sure that they could develop a way to have several bitstreams running concurrently to teh disk and have several heads reading/writing on the same disk, same side/etc, but I am sure that with each bitstream you use, the management of them gets increasingly complicated.

Also, if you use a parking lot analogy yuo can also see how one thing can outpace the other. Initially we were all driving bombers and could only fit a certain ammount of cars in there, but as time went by, we changed cars to smaller, and optimized the parking layout. We put in an extra attendant and people were able to get in and out faster.

Now later, you put in a parking garage, add another attendant and go from there. You are not fitting 10 times the people in there, but your attendants only went from 2 to 3, so your performance was trounced by your storage capacity. Add to the mix that now, although we have faster attendents, with quicker cash registers, we are parking our cars vertically. I wonder if there will be another ratio lag between performance and storage or if, as teh resident techie here informed us, that we actually catch up (all the cars parking there get E-Z Pass... ;) )

The analogy can also be extended to water storage and pump capacity and a number of different associations, but the end result is the same. Increasing capacity is easier than increasing the performance on a physical read-write drive.



I am just wondering what is happening with flash-drives and whether that teoretical 100G will ever be possible within a CF cards footprint. If that is the case, parallel read/write for dozens of these cards may be the next step in mass storage on our machines.

Imagine having 20 of them arranged in your machine in a RAID array! 1 goes bad, an LED indicates which one, you replace that one alone!

Of course, for this to become a reality, the prices will have to drop significantly, but I am anxious to see how this all plays out in the next 10 years...
 

yyrkoon

Distinguished
May 8, 2006
387
0
18,780
I’ve been telling people this for years.

Everyone talks about CPU and GPU power and the latest high speed ram. No one ever talks about hard drives. So I'll say it again.

Instead of going out and buying that 500 gig western Digital... Go buy 2 250 gig drives and raid stripe them.

I now have 3 sets of of Western Digital raptor drives:

2 X 36
2 X 72
2 X 150 (sata2)

I have 7 systems at home and no matter what I put in a box, running 2 drives in stripping has by far the greatest impact to perf.


PS. I remember buying that Maxtor 40 meg hard drive for my old 386 16 and it was a beast!!!!

Maybe you edit video ? RAID will not improve typical desktop performance that_much. I have 4xRAID0 running on my system here, and other than for seriously HDD intensive tasks, it offers no speed increase. Mostly I only see improvements when loading saved games, or game levels, editing Video, and extracting large archive files (oh, and seeing Sandra bench 4X barracudas(ATA 100) at 132MB/s, and 7 ms random access times is neat too). Of course, if you want to shave some time off your OS boot times, RAID0 can also come in handy, but unless you're running 0+1, or perhaps RAID10, you're playing with fire. *shrug*

Don't get me wrong, I'm a serious hardware buff, and love the idea of any type of RAID, but unfortunately, except for the occasional desktop application, it doesn't really offer much of a performance boost. I've even noticed that putting my swap file on a 4x RAID 0 array made very little(if any) impact on overall system performance. That is, a noticeable performance difference, of course.

All this being said, I still want to get a good Areca RAID controller, and run a serious 4-8x RAID0 Array (before swapping to RAID 5, or 6) just to see how fast this array would perform. Call it the hardware "kid" in me, that still keeps trying to eek out every last possible ounce of performance . . .

P.S. I also had that 40MB HDD in my 386 sx 25 (1993), and still have the drive that replaced it, a 80MB Maxtor, and it still works !
 

wicketr

Distinguished
Nov 27, 2006
9
0
18,510
Please oh please add the noise/audible level of hard drives to your hard drive chart. That would be very helpful.

I care more about noise level than getting that 1/1000th performance increase over another hard drive.
 

tleavit

Distinguished
Jun 28, 2006
145
0
18,680
Maybe you edit video ? RAID will not improve typical desktop performance that_much. I have 4xRAID0 running on my system here, and other than for seriously HDD intensive tasks, it offers no speed increase. Mostly I only see improvements when loading saved games, or game levels, editing Video, and extracting large archive files (oh, and seeing Sandra bench 4X barracudas(ATA 100) at 132MB/s, and 7 ms random access times is neat too). Of course, if you want to shave some time off your OS boot times, RAID0 can also come in handy, but unless you're running 0+1, or perhaps RAID10, you're playing with fire. *shrug*

Don't get me wrong, I'm a serious hardware buff, and love the idea of any type of RAID, but unfortunately, except for the occasional desktop application, it doesn't really offer much of a performance boost. I've even noticed that putting my swap file on a 4x RAID 0 array made very little(if any) impact on overall system performance. That is, a noticeable performance difference, of course.

All this being said, I still want to get a good Areca RAID controller, and run a serious 4-8x RAID0 Array (before swapping to RAID 5, or 6) just to see how fast this array would perform. Call it the hardware "kid" in me, that still keeps trying to eek out every last possible ounce of performance . . .

P.S. I also had that 40MB HDD in my 386 sx 25 (1993), and still have the drive that replaced it, a 80MB Maxtor, and it still works !

I have a lot of computers (at home, I manage 250 at work) right now that are pretty much the same spec such as:

P4 2.6 (533)
p4 2.8 Prescott
p4 3.0 Prescott
p4 3.0 Prescott
p4 3.4 Prescott
AMD 3500+
AMD 4400+ X2
Core 2 Duo 2.6


All with similar hardware (same Asus MB, generic Kingston ram and such).

One may have the raid, one may have say a 200 gig Western digital Sata drive.

The difference in doing any task in Windows is greatly improved. Simple things such as opening Itunes (with 60 gigs of MP3's) is significantly improved. I actually find it hard to use the non-raid PC's since the delays the HD cause are noticeable.


Real life example
When I backup the 60 gigs in mp3's from one to another, I can fire up WOW on the AMD 4400+ with absolutely no hard drive lag (hell, I cant even tell its crunching the data). On the Core 2 duo with 4 gigs or ram but only a 250 gig sata drive, the PC is severely lagged and cannot be used for much until its done. This applies to all of my PC's. The raid drives clearly perform much better.


I think one does not really see the benefit until you sit them next to each other and just use the PC's.
 

tleavit

Distinguished
Jun 28, 2006
145
0
18,680
SCSI or Sas?

SCSI.

Well, for the same price (or less with the SCSI card) you could have probably bought 2 Western digital 150gig Sata2 rapters and blow that drive away in striping.

Scsi is great on my servers, I would never run it at home.

I just rolled out a Dell Poweredge NAS with Storage server X64 enterprise. The thing has 8 300 gig SAS drives in Raid 0+1 (1 terrabye of data). The thing smokes but I still wouldnt put those drives in on my gamming PC. Im not running DFS at home and 250 users dont use it to file server :)
 

yyrkoon

Distinguished
May 8, 2006
387
0
18,780
I have a lot of computers (at home, I manage 250 at work) right now that are pretty much the same spec such as:

P4 2.6 (533)
p4 2.8 Prescott
p4 3.0 Prescott
p4 3.0 Prescott
p4 3.4 Prescott
AMD 3500+
AMD 4400+ X2
Core 2 Duo 2.6


All with similar hardware (same Asus MB, generic Kingston ram and such).

One may have the raid, one may have say a 200 gig Western digital Sata drive.

The difference in doing any task in Windows is greatly improved. Simple things such as opening Itunes (with 60 gigs of MP3's) is significantly improved. I actually find it hard to use the non-raid PC's since the delays the HD cause are noticeable.


Real life example
When I backup the 60 gigs in mp3's from one to another, I can fire up WOW on the AMD 4400+ with absolutely no hard drive lag (hell, I cant even tell its crunching the data). On the Core 2 duo with 4 gigs or ram but only a 250 gig sata drive, the PC is severely lagged and cannot be used for much until its done. This applies to all of my PC's. The raid drives clearly perform much better.


I think one does not really see the benefit until you sit them next to each other and just use the PC's.

Well, I've been building systems for about 13 years, and have 4 right here in front of me working, one of which has had a RAID0 array as the boot partition for 2-3 years (ABIT KT7A-RAID 100 board, AMD2000+ CPU). Another system sitting right next to it, is an ABIT NF7-S2G, AMD 3200+ XP, 1GB DDR 400, with a SATA drive, and I can tell you, it is faster than the older system running RAID. However, there is an architecture difference between systems, and the older system does use PC-133 memory vs. DDR in dual channel mode. Also the older system is a PCI 2.1 based system, vs the PCI 2.2 / PCI-E architecture of the newer system (even though the newer board does not have any PCI-E slots).

Granted, I've never been a big fan of WD, and their Raptors, as I can not see paying 5x as much per GB for marginally better performance(recently I made a post on THFs here, using an XP pro software 4xRAID0 mixed matched Barracudas vs 4xRAID0 36GB Raptors, and the results were not really all that favorable for WD). Still, if I were made of money, I would probably consider buying a few Raptors, and slap them together in RAID0 on an Areca controller of some sort, and have some serious fun. Although I could be wrong, I seriously doubt the claims the guys here at TH about the Raptors doing 85MB/s, at least, not 85MB/s sustained.

As for all servers here, we pretty much stick with Ultra 160 SCSI drives, or better, although, if SAS equipment would drop in price, I would consider that drive type.

Anyhow, sure, copying large amounts of files from/to one RAID array, to another is going to be faster. There is a huge difference even with my software RAID array here, and thats why I have one, but as far as making the system faster, well, I guess that all depends on what you do, and is really subjective. Normally however, I personally do not copy large amounts of files, while playing a games, but I do edit video a lot (sometimes), and it can make a world of difference. Also, compiling programs under Linux, or Windows, on a RAID0 array seems to make a big difference (at least in my experiences).

One thing I would really love to do here at home, would be putting together a 10GbE Network, running massive amounts of disks on an iSCSI Target in RAID, and watching as copying huge files happens in a matter of seconds, but alas, funds dictates otherwise :(
 

tleavit

Distinguished
Jun 28, 2006
145
0
18,680
I have a lot of computers (at home, I manage 250 at work) right now that are pretty much the same spec such as:

P4 2.6 (533)
p4 2.8 Prescott
p4 3.0 Prescott
p4 3.0 Prescott
p4 3.4 Prescott
AMD 3500+
AMD 4400+ X2
Core 2 Duo 2.6


All with similar hardware (same Asus MB, generic Kingston ram and such).

One may have the raid, one may have say a 200 gig Western digital Sata drive.

The difference in doing any task in Windows is greatly improved. Simple things such as opening Itunes (with 60 gigs of MP3's) is significantly improved. I actually find it hard to use the non-raid PC's since the delays the HD cause are noticeable.


Real life example
When I backup the 60 gigs in mp3's from one to another, I can fire up WOW on the AMD 4400+ with absolutely no hard drive lag (hell, I cant even tell its crunching the data). On the Core 2 duo with 4 gigs or ram but only a 250 gig sata drive, the PC is severely lagged and cannot be used for much until its done. This applies to all of my PC's. The raid drives clearly perform much better.


I think one does not really see the benefit until you sit them next to each other and just use the PC's.

Well, I've been building systems for about 13 years, and have 4 right here in front of me working, one of which has had a RAID0 array as the boot partition for 2-3 years (ABIT KT7A-RAID 100 board, AMD2000+ CPU). Another system sitting right next to it, is an ABIT NF7-S2G, AMD 3200+ XP, 1GB DDR 400, with a SATA drive, and I can tell you, it is faster than the older system running RAID. However, there is an architecture difference between systems, and the older system does use PC-133 memory vs. DDR in dual channel mode. Also the older system is a PCI 2.1 based system, vs the PCI 2.2 / PCI-E architecture of the newer system (even though the newer board does not have any PCI-E slots).

Granted, I've never been a big fan of WD, and their Raptors, as I can not see paying 5x as much per GB for marginally better performance(recently I made a post on THFs here, using an XP pro software 4xRAID0 mixed matched Barracudas vs 4xRAID0 36GB Raptors, and the results were not really all that favorable for WD). Still, if I were made of money, I would probably consider buying a few Raptors, and slap them together in RAID0 on an Areca controller of some sort, and have some serious fun. Although I could be wrong, I seriously doubt the claims the guys here at TH about the Raptors doing 85MB/s, at least, not 85MB/s sustained.

As for all servers here, we pretty much stick with Ultra 160 SCSI drives, or better, although, if SAS equipment would drop in price, I would consider that drive type.

Anyhow, sure, copying large amounts of files from/to one RAID array, to another is going to be faster. There is a huge difference even with my software RAID array here, and thats why I have one, but as far as making the system faster, well, I guess that all depends on what you do, and is really subjective. Normally however, I personally do not copy large amounts of files, while playing a games, but I do edit video a lot (sometimes), and it can make a world of difference. Also, compiling programs under Linux, or Windows, on a RAID0 array seems to make a big difference (at least in my experiences).

One thing I would really love to do here at home, would be putting together a 10GbE Network, running massive amounts of disks on an iSCSI Target in RAID, and watching as copying huge files happens in a matter of seconds, but alas, funds dictates otherwise :(


Ya, we put that power edge in to replace a Iomega NAS device that we had that was using 3 drives configured for raid 5 in windows (software). That device raid was amazingly slow. I would never suggest anyone use software RAID.
 

icepop456

Distinguished
Nov 27, 2006
19
0
18,510
I’ve been telling people this for years.

Everyone talks about CPU and GPU power and the latest high speed ram. No one ever talks about hard drives. So I'll say it again.

Instead of going out and buying that 500 gig western Digital... Go buy 2 250 gig drives and raid stripe them.

I now have 3 sets of of Western Digital raptor drives:

2 X 36
2 X 72
2 X 150 (sata2)

I have 7 systems at home and no matter what I put in a box, running 2 drives in stripping has by far the greatest impact to perf.


PS. I remember buying that Maxtor 40 meg hard drive for my old 386 16 and it was a beast!!!!

You do realize that running a striped array as you described is twice as likely to fail, right? I agree with your idea of performance increase, but for most people, losing a drive could be the end of their pictures, music, etc. Yes, we all should have backups, but we also know they are not done. Therefore, I hardly ever recommend anyone stripe hard drives for performance increases. I personally run the following : (A1||A2) + (B1||B2) for my array. That is, two drives are mirrored to form one A drive and the same for B. Then A+B are stripped. Not ideal, but I can suffer 1 and, in some situations, 2 drive failures without data loss. Not ideal for storage, but I get 100MB/s read/write which is good for me. However, not everyone can afford 4 drives with 50% storage efficiency.
 

tleavit

Distinguished
Jun 28, 2006
145
0
18,680
I’ve been telling people this for years.

Everyone talks about CPU and GPU power and the latest high speed ram. No one ever talks about hard drives. So I'll say it again.

Instead of going out and buying that 500 gig western Digital... Go buy 2 250 gig drives and raid stripe them.

I now have 3 sets of of Western Digital raptor drives:

2 X 36
2 X 72
2 X 150 (sata2)

I have 7 systems at home and no matter what I put in a box, running 2 drives in stripping has by far the greatest impact to perf.


PS. I remember buying that Maxtor 40 meg hard drive for my old 386 16 and it was a beast!!!!

You do realize that running a striped array as you described is twice as likely to fail, right? I agree with your idea of performance increase, but for most people, losing a drive could be the end of their pictures, music, etc. Yes, we all should have backups, but we also know they are not done. Therefore, I hardly ever recommend anyone stripe hard drives for performance increases. I personally run the following : (A1||A2) + (B1||B2) for my array. That is, two drives are mirrored to form one A drive and the same for B. Then A+B are stripped. Not ideal, but I can suffer 1 and, in some situations, 2 drive failures without data loss. Not ideal for storage, but I get 100MB/s read/write which is good for me. However, not everyone can afford 4 drives with 50% storage efficiency.

True but I only buy Western Digital and contrary to things I've read on these forums, I have never once had a problem with Western Digital :)

When I started in this industry I worked at a place that rolled out 2000 computers in our state. Compaqs mainly with all varieties of drives. When we saw a Western Digital it was a god send because we were always being dispatched to replace Maxtor and Seagate drives. We lived through the "big foot" drive, had to replace 100 of those. about 7ish years ago we lived through a Maxtor recall (20 gig drives if I remember right) where we had to proactively replace 200 because they dropped like flys. We probably rolled out a total of 5000 PC in 5 years and I don’t think we had one WD drive crash but it was almost common on Maxtor/Seagate/Deathstar’s.

I've had my first 2 x 37 gig raid going for 4ish years now with no problem.

We'll I did have one problem with raids and if I remember right it was on an Early Adus board that had flawed firmware that kept breaking mirrors.

But note:

I do not use those raid drives for my data storage, I don’t trust them enough. I run them either on my dedicated gamming machine (that can be rebuilt in an hour) or as partitions that I rip my digital home videos to. I run the 3rd on my htpc for capture but store the data on normal drives. I have about 2 terabytes worth of storage at home.