Sign in with
Sign up | Sign in

The Dirty Secrets Of SSD Benchmarking

2011 Flash Memory Summit Recap: Tom's Hardware Represents
By

Where is Steady State?

There are a couple of dirty secrets that most Tom's Hardware readers know, but few mainstream buyers would have ever had occasion to hear. The first has to do with performance fresh out of the box versus steady-state performance. SSD manufacturers prefer that we benchmark drives the way they behave as soon as you buy them because solid-state drives slow down once you start using them. If you give an SSD enough time, though, it reaches a steady-state level. At that point, its benchmark results reflect more consistent long-term use. In general, reads are a little faster, writes are slower, and erase cycles take place as slowly as you'll ever see from the drive.

There’s been a movement within the flash storage community to standardize performance analysis to focus on steady-state. A lot of conference presenters kept stressing this point. However, they left out one important detail: there isn’t one single steady state. Instead, you might see 4 KB random writes stabilize at one point, and 128 KB sequential writes stabilize somewhere else. You can even get more granular and figure out where 128 KB sequential writes hit their steady-state level in bursty and sustained transfers.

Source: MicronSource: Micron

The main takeaway was that there isn't one moment in time where we can declare, "this drive just reached its steady-state performance level; we're ready to test it." You could benchmark a single SSD for an entire month and still not have its complete performance profile worked out. What you write, how fast you write it, how much you write, the I/O workload from the previous days and weeks and so on all change how the SSD performs.

We prefer to focus on the overall picture, including the main points that help you make an informed purchase. It’s not that other benchmarks don’t matter, but we’re trying to present SSD performance in manageable bites that don’t overwhelm. That’s why we focus on a consumer-oriented steady state reached within a reasonable period, which you can read more about on page four of Second-Gen SandForce: Seven 120 GB SSDs Rounded Up.

Consumer vs. Enterprise Benchmarking

Almost every SSD's specification sheet cites performance based on when the drive is brand new, which we've already established is only representative of what you get for a brief period. However, manufacturers blur the performance picture even more by generating all of their data using high queue depths. We don't really take issue with that in the enterprise space, because SSDs may very well encounter high queue depths when they're getting hammered by online transaction processing or some other, similar workload. Solid-state drives on the desktop simply don't encounter those same queue depths, though. As a result, most performance guidance is overstated.

We can't blame the SSD industry for representing performance this way. After all, vendors are responsible for shining the best possible light on their products, and high queue depths are, in fact, the best way to saturate the multi-channel architectures employed by SSD controllers, yielding the best performance. More important is that end-users realize they won't see those same aggressive figures on their own desktops.

This is something we’ve tried to emphasize as of late, giving you a more realistic perspective of everyday performance. And we can promise you that there are more than a couple of SSD vendors unhappy with us for running our tests using queue depths derived from real-world traces. But we answer to you, not them. And so, the differences between SSDs are fewer in our results than you might otherwise expect from comparing spec sheets. The benefit is that you don't actually have to buy the latest or most expensive SSD in order to enjoy a substantial performance boost.

Display all 13 comments.
This thread is closed for comments
  • 0 Hide
    compton , August 16, 2011 4:31 AM
    Is there a way for a user to record their own week of disk access, and in so doing make their own Storage Bench (with the assistance of some other software)?
  • 3 Hide
    Anonymous , August 16, 2011 4:44 AM
    Still very surprised hybrid hard disks like the Momentus XT aren't getting more R&D. If they included even half the capacity of a mainstream HD and a quarter the capacity of a mainstream SSD, they'd have an amazing product. Just think: a 500GB platter paired with 32GB SSD for, what, $150? I'd buy that in a jiffy.

  • -2 Hide
    chovav , August 16, 2011 8:48 AM
    caywen - I don't think I agree. This approach means you will lose both SSD and hard drive if one fails. Second, a small SSD like that will only be useful for caching, which we all saw until now isn't worth much.

    lastly, wanting to upgrade will mean having to upgrade both, instead of one. I think this is one of the reasons why the XT was never so popular...

    O.T - nice read. It did feel a bit as though it was stopped in the middle..
  • 2 Hide
    mkrijt , August 16, 2011 11:34 AM
    chovavIt did feel a bit as though it was stopped in the middle..


    That was exactly what I was thinking. I was looking for the "next page" button but couldn't find it. I was like wtf? went back to the first page to look at the index finding it really ended there :( 
  • 4 Hide
    cknobman , August 16, 2011 1:09 PM
    mkrijtThat was exactly what I was thinking. I was looking for the "next page" button but couldn't find it. I was like wtf? went back to the first page to look at the index finding it really ended there


    LOL me too!!!

    For me personally I am willing to drop up to $120 on a SSD but that is the breaking poing for me and I am not willing to settle for anything less than 120GB due to the performance drop in smaller SSD's and also how much space I need for my OS and other apps (and no I am not counting data like movies or pics).

    The main thing holding me off on purchasing a SSD right now is the lack of confidence in reliability. The bugs in Intel, Sandforce, and whatever controller crucial uses in the m4 makes me worried. Just looking at articles here and user reviews on NewEgg was enough for me to get gun shy and wait. There are few things I hate more than having to setup my system and even though I can ghost my boot drive there will always be some loss in a drive failure and that is just something I want to avoid if I can. Unfortunately it just seems like owning a SSD right now leaves too high a risk of drive failure. Plus it does not help reading how manufacturers refuse to comment or give any real hard data on reliability.
  • 1 Hide
    nforce4max , August 16, 2011 10:21 PM
    What about page file related write attributed to normal use? It is very easy to see more than 10gb a day worth of writes depending on the apps used.
  • 0 Hide
    razvanz , August 17, 2011 7:32 AM
    nforce4max I agree with you. I haven't seen any SSD tests using bittorent. How does a SSD fare against a HDD in bittorent usage. I think bittorent will easily overwhelm the read/write cycles of a SSD.
  • 0 Hide
    JamesAbel , August 17, 2011 2:56 PM
    On the surveys on "willingless to pay for SSDs" - has this ever been done on someone that has TRIED an SSD? Reading about a benchmark is different than experiencing it. On my SSDs, I see huge benefits in things like app load time vs. an HDD. Also, SSDs are much faster vs. HDDs when multiple apps are running and causing lots of head seeks (drive thrash). Give someone an SSD for a month and then give them back an HDD - I think many won't be so willing to switch back (and be willing to pay from there on out).
  • 0 Hide
    Mark_Alberta , August 17, 2011 4:40 PM
    I have been testing various SATA SSD's for recording CCTV footage for several months now. The results have been disappointing when compared to mechanical hard drives. First off, unless I set the maximum file size to 85% or less of the drive's capacity, instead of 95% with say, a WD Caviar Black, the drive would over write the data 3-4 times and then suddenly become undetectable on the system until I reboot it. Not a good thing when considering the purpose of the unit. This has occurred on three different manufacturer's drives, so it's obviously not a one-off issue. Another interesting problem that can up was on the 40GB OS drive, the detected volume dropped from 37GB to roughly 24GB. I reformatted the drive, re-installed the OS and everything returned to normal. I am going to continue to try and perfect a system using SSD's, but it is a lot more of a headache that the simplicity of plugging in a 3.5" mechanical drive and moving on.
  • 0 Hide
    drwho1 , August 17, 2011 6:54 PM
    I would be willing to pay from $150 to @200 top for a 500GB SSD.
    When prices and size ratio meet this price standards then I would take a serious look into SSD's.

    Have in mind that current 2TB hard drives are under $100 dollars (true they are green drives). But when a SSD per GB cost 10 to 20 times as much a hard drive is just not right to me.
  • 0 Hide
    kyuuketsuki , August 18, 2011 2:57 AM
    I maintain that simply looking at $/GB of SSD versus HDD and deciding based on that is foolish. Getting an SSD isn't about storing reams of data. If you need to store a lot of data that is accessed infrequently (and/or usually accessed sequentially, like movies/music/pictures/etc.) then you store that on an HDD, hands down no questions asked. But for applications and the operating system where HDD's incredibly slow random read/write and long seek times cause thrashing anytime anything remotely demanding on the storage subsystem happens, the value of an SSD is more than worth it to anyone who values performance.

    It's no different than spending $200+ on the latest Sandy Bridge CPU instead of $100 on a perfectly capable Phenom. Or spending more than twice as much for performance RAM over value RAM (even though the speed benefits of performance RAM are practically non-existent compared to the difference between an SSD and an HDD). Or getting a dGPU rather than using the iGPU that comes with your CPU.

    Plus, the value of an SSD climbs when you consider that you're almost guaranteed to be able to carry it over to your next build, and the one after that. Probably even the one after that.
  • 0 Hide
    ojas , August 20, 2011 11:50 AM
    Nice article.

    I personally think that the SSDs will become really attractive when a 120GB SSD costs about $150, though i think 1$/GB should be the target. Reliability too. I'm holding off a SSD purchase for a 2 or 3 years, when their statistics would be more developed and their price and bugs would be lowered.
  • 0 Hide
    Anonymous , August 23, 2011 4:47 AM
    I have been waiting for 4 years to prices to fall down, but so far they main improvements seem to be mainly on speed.
    The minimum size for me is 120-128 Gb but that is still tight with my current system and programs, so the right size would be 256 Gb which costs as much as whole Desktop.