Why You Need a Reliable Storage Benchmark
Why You Need a Reliable Storage Benchmark
What's this
Most of us feel cheated, even angry, when we discover that trusted sources fail to provide us the entire story. Sometimes bias and self-interest are factors, while other times there is simply a lack of complete information. It’s the individual’s responsibility to demand the whole story. Buyers need to put their journalist hats on and go digging for the real scoop. Unfortunately, when it comes to storage performance, this seldom happens.
For several years, the enterprise computing world has relied on a handful of benchmarking tools to assess storage latency, throughput, and I/O operations per second (IOPS). The open source program Iometer, originally designed by Intel, is one of the most prominent of these tools. And Iometer is very good at what it does. However, it doesn’t deliver the whole story, and without all the information, IT decision makers are open to misinterpreting test data and making poor storage solution choices.
This paper discusses a non-profit industry group called the Storage Performance Council, or SPC (www.storageperformance.org), and its complementary role in benchmarking storage devices. Along the way, we hope to give you the real scoop and provide more insight into how business-class storage testing can and should be done.
Best Practices
Storage testing looks easy. Throw a drive or an array on a tech bench, run some tests, and voila—bulletproof results. Occasionally, that process works. All too often, though, truly precise and dependable results can be elusive. We encountered a case recently where a certain vendor’s hard drive would perform optimally for a brief period, but the longer that drive stayed under load, the hotter it got. When the temperature got high enough, the drive would automatically throttle down performance in order to protect itself. Clearly, this is not the kind of result that a buyer wants to see.
The question is whether that throttling behavior would show up in benchmark tests, and the answer is more complex than you might think. To help clarify the issues related to storage device testing, we’ve put together a list of best practices that can help storage buyers dig below superficial test data and determine whether a given drive behaves as it should under environmental circumstances that are truly relevant to the prospective owner.
1. Ask vendors to have product performance claims validated by a proven, independent third party auditor.
Right or wrong, many people are inherently distrustful of vendor-generated test data. This is why Consumer Reports magazine continues to thrive. In the same vein, it’s not always obvious when some supposedly objective reviewers or their media outlets may have been unduly biased. Ideally, you want test data to come from a fully neutral, independent organization. Even if the organization won’t supply results directly, a vendor should be willing to submit products for testing and utilize the validation process either to further refine the product or vouch for its performance. We see this everywhere from graphics cards in NVIDIA’s SLI program to power supplies certified through 80 PLUS. Storage should be no different.