SPEC and SIPEW 2008 – Benchmarking for Professionals

Many people are familiar with the Standard Performance Evaluation Corporation (SPEC), a non-profit organization with the goal of providing relevant and realistic standard benchmarks. SPEC is based in Warrenton, Virginia (online at http://www.spec.org) and currently has more than 60 members. These include renowned universities and many big players in the hardware industry who want to help shape benchmark software that will eventually be used to evaluate and assess products. The organization does not make recommendations, but it provides performance results to the public.

SPEC Divisions

SPEC is divided into three divisions that take care of system level benchmarks, high performance computing and graphics benchmarking. The portfolio includes benchmarking tools for processors (SPEC CPU2006, CPU2000), professional graphics performance (SPECviewperf and SPECapc workloads), high performance computing (SPEC MPI2007), client/server benchmarks based on Java (SPECjAppServer, SPECjbb, SPECjms, SPECjvm), different server requirements, and a benchmark to determine power efficiency (SPECpower_ssj20). None of these are consumer benchmarks; they are aimed at comparing performance in professional and enterprise environments.

SPEC provides tools that enable a meaningful differentiation of system solutions on a fair basis, while allowing evaluators to focus on individual system characteristics. Benchmarks are thus based on popular applications and industry standard software that has already been ported to common platforms. Prior to running benchmarks, source code has to be compiled on the target system, using optimized compilers and specific settings to reach the best performance for each target environment.

While the SPECapc and SPECviewperf graphics benchmarks can be downloaded for free—we’ve been using SPECviewperf to evaluate OpenGL graphics cards—the CPU and system benchmarking solutions have to be ordered. The cost for these varies from $50 and $2,000, depending on the development efforts for SPEC. There are huge discounts for non-profit and educational organizations, resulting in a $50 to $900 price range.

SPEC Results are the Charts for Pros

Members are encouraged to submit their benchmark results to SPEC, which will review the submissions and publish the results on its website (see SPEC CPU2006 results as an example, and the results submission page for details). Since SPEC is the most renowned institution for professional benchmarking, companies such as AMD or Intel are eager to publish the results of their best performing solutions on SPEC. As a result, these numbers can be considered as a public and highly recognized top list for decision makers.

We visited SPEC’s so-called SIPEW workshop in Darmstadt, Germany, where members discussed the issue of scheduling in server farms, and the power efficiency benchmark SPECpower_ssj2008.