Industry Group Accuses Nvidia of Twisting TPCx-BB Benchmark Results

The Transaction Processing Performance Council (TPC), an industry consortium that develops performance benchmarks for servers and data centers and includes names like AMD, Intel, and IBM, has accused Nvidia of violating the TPC’s Fair Use Policy. To compare the performance of its GPUs against other machines, Nvidia presented results from workloads derived from official TPC benchmark workloads and compared them to official TPC results, a violation of the group's fair use policy.

At its GTC Fall 2020 virtual event last year, Nvidia presented a paper called State of RAPIDS: Bridging the GPU Data Science Ecosystem. In the paper, Nvidia demonstrated the advantages of its RAPIDS suite of GPU-accelerated software libraries for data science workflows over competing hardware and software platforms.  

Among other things, Nvidia compared results of its GPUs in workloads similar to those from the TPCx-BB, a TPC big data analytics benchmark that measures the performance of Hadoop-based Big Data systems. The benchmark executes 30 regularly performed analytical queries in the context of retailers with physical and online stores. The queries use regular SQL for structured data and machine learning algorithms for semi-structured and unstructured data. The TPCx-BB benchmark workloads are precise and subject to TPC's Fair Use Policies. 

The GPU giant claimed that its platform "outperformed by nearly 20x the record for running the standard big data analytics benchmark, known as TPCx-BB." However, Nvidia didn't run the workloads designed by TPC and used similar workloads instead, which violates TPC's policies and makes such comparisons invalid and generally unfair. However, Nvidia did mark the version of the TPCx-BB benchmark it used as 'not official.' 

"The TPC actively encourages publicizing of TPC results by all organizations, including the press, market researchers, financial analysts and non-profit organizations," said Mike Brey, chairman of the TPC Steering Committee. "However, to ensure that users and readers of TPC results are given a fair and complete representation of TPC data, the TPC requests that all users follow the Fair Use rules, outlined in TPC policies, when publishing or publicizing results." 

The TPC said it was working with Nvidia to correct the issue. 

TPC is led by companies that produce CPUs, servers, and business software, including AMD, IBM, and Intel. It also includes cloud data center operators like  Alibaba, Microsoft, and Oracle. Nvidia is not a member of the TPC.