Sign in with
Sign up | Sign in

The SPC and Its Testing

Why You Need a Reliable Storage Benchmark
By
Brought to you by What's this

The Storage Performance Council (SPC, www.storageperformance.org) was founded in 1998 as a non-profit corporation with a fairly straightforward mission: “...to define, standardize, and promote storage subsystem benchmarks as well as to disseminate objective, verifiable performance data to the computer industry and its customers.” In other words, there was a vacuum in the industry wherein no independent, reliable storage benchmarking existed, and the SPC has worked ever since to fill that void.

Today, the SPC hosts over four dozen member companies on its roster, including controller heavyweights LSI, Marvell, SandForce, and Promise Technology along with drive manufacturers Seagate, Samsung, Fujitsu, Toshiba, Hitachi, and Western Digital. Solutions providers Dell Compellent, EMC, HP, IBM, NetApp, and others also belong to the Council.

According to the SPC, its portfolio of benchmarks remains the only industry standard suite for producing comparative data in regard to storage performance, price-performance, and energy use. The tests are built to be device-, vendor-, and platform-agnostic. Anyone who is a member of the SPC can run the organization’s benchmarks, but publication and acceptance of the results within the industry requires an SPC audit. This is to ensure accuracy and authenticity.

The SPC-1 Benchmark Series

The SPC-1 benchmark group focuses on I/O workloads typically generated by OLTP applications. Specifically, the SPC-1 and SPC-1E tests are built for complex, fully configured storage systems while SPC-1C and SPC-1C/E are geared more for individual storage devices, including drives, controllers, and enclosures.

The SPC Benchmark-1 specification was first used in October 2001, but it wasn’t until September 2009 that the Energy Extension clause found in SPC-1E became official. Those who want to plow through the complete 120-page spec can find it at http://www.storageperformance.org/specs/SPC-1_SPC-1E_v1.12.pdf, but the nut of it comes down to synthetic I/O loads generated by hypothetical business users (Business Scaling Units, or BSUs) and either the source or destination of that business data (Application Storage Units, or ASUs). The BSU creates the workload, and the ASU is the abstracted arena in which that workload operates.

Because SPC-1 seeks to replicate performance in a mission critical application scenario, the workload divides into three ASUs: Data Store (I/O command traffic), User Store and Log/Sequential Write. The specifics get much more complex from there, but the bottom line is that each ASU involves multiple streams of random and sequential read/write data. The SPC feels its methodology represents the type of load commonly placed on an enterprise storage subsystem. The SPC-1E Energy Extension compares idle power consumption against consumption at various levels of I/O traffic, ultimately arriving at IOPS/watt and annual energy use estimates. SPC reports already include some price/performance analysis, but having energy efficiency data alongside this presented in a standardized fashion will hopefully help edge the industry toward greater overall energy efficiency awareness.

SPC-1C was released in April 2008; the SPC-1C/E Energy Extension was added in May 2009. The methods for these device-level tests somewhat mirror the SPC-1 system tests, right down to using the same weighting of I/O traffic.

The SPC-2 Benchmark Series

The SPC-1 benchmark group focuses on I/O workloads typically generated by random I/O applications—the sort of loads commonly found in transactional environments. In contrast, SPC-2, which debuted in December 2005, squarely aims at workloads involving mass quantities of simultaneous sequential transfers. Rather than focusing on IOPS, the emphasis here is on real time processing throughput and meeting data persistence (protection) requirements. Sample workloads are constructed specifically to address three key application types: large file processing, large database queries, and video on demand.

As in the SPC-1 series, there is also a SPC-2C specification for storage devices rather than systems. Interestingly, though, there are currently no Energy Extensions offered in the SPC-2 series.