Intel Stands Behind Controversial Tests That Favored Its CPU Over AMD's

It's a basic rule of benchmarking. When you're comparing two competing products, you need to make all the testing conditions -- especially the settings -- as similar as possible. Otherwise, you're just comparing apples to crab apples, which is what may have happened in a recent benchmark report that showed Intel's new Core i9-9900K outperforming AMD's Ryzen 7 2700X in several games.  

As part of its Coffee Lake Refresh launch, Intel paid test firm Principled Technologies to benchmark its new processors and compare them to chips in AMD's current lineup (PDF). However, Hardware Unboxed spotted a few test conditions that obviously could skew the results in favor of Intel's processor, including using a less-capable CPU cooler on AMD's chip, overclocking the RAM on Intel's platform only, and disabling half the cores on an AMD Ryzen processor.

We reached out to Intel for comment and got the following reply:

"We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world’s best gaming processor. Principled Technologies conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third-party testing in the coming weeks."

As you can see, Intel is standing by its assertion that the Core i9-9900K is the "world's best gaming processor," and given its specifications and Intel's current dominance in absolute gaming performance, the statement is likely true. However, Intel doesn't go as far as a full-throated endorsement of the report, instead saying the results are "consistent with what we've seen in our labs," which is an important distinction.

We also reached out to Principled Technologies for comment but didn't receive a response as of press time.

There are few absolute truths, but the old mantra of "never trust vendor's benchmarks" is certainly one of them. That means today's findings that the benchmarks could be flawed isn't entirely surprising. We've seen misleading reports in the past from almost every vendor, but it is concerning that this report comes during Intel's 10-day preorder window.

Here are some of the controversial results in Principled Technologies' report.

Principled Technologies' benchmarks pit the Core i9-9900K, Intel's new flagship eight-core processor for mainstream systems, against AMD's mainstream desktop Ryzen 7 2700X. The 19 game benchmarks in the report paint a very convincing picture of the Core i9-9900K's total dominance over the Ryzen 7 2700X in gaming, but a deeper look at the system settings revealed a few questionable, and one downright unbelievable, configuration that could skew the results in Intel's favor.

The missteps begin with system memory. Principled Technologies used the XMP profile, which automatically assigns tight timings for the Corsair Vengeance DDR4-3200 memory (running at DDR4-2666) installed in the Intel system. In contrast, the company set the Ryzen 7 2700X's memory configuration, which was fully populated with four double-sided DIMMs, at the stock DDR4-2933. This setup allows the system to assign timings for the memory, which often leads to suboptimal memory performance.

As we've proven time and again, AMD's Zen architecture is extremely sensitive to memory performance, particularly in games. As such, even small variations in settings result in improved gaming performance with Ryzen processors, meaning this mismatch could give Intel at least some advantage in the gaming tests.

Principled Technologies also used the stock Ryzen 7 2700X Wraith Prism cooler with the AMD system but installed a premium Noctua NH-U14S cooler on the Intel processor. Again, as we've proven in the past, improved cooling benefits both AMD and Intel's chips by allowing the processors to operate at their Boost frequencies more frequently, and then maintain the heightened clock speeds for longer periods of time.

Slight differences in cooling solutions can have an impact on the test setup. In this case, the Intel system could expend more waste heat than the AMD system, which would provide it with an advantage. Intel's new K-series processor doesn't come with a stock cooler, so the company had to use a third-party solution, but sound testing methodology dictates that coolers for both systems should be of equal quality. That didn't happen here.

Perhaps the most damning admission involves AMD's Game Mode feature that AMD created for its Threadripper processors. Game mode essentially disables half of the processors' available cores to circumvent the intricacies of the Threadripper architecture. Those same principles don't apply to the mainstream Ryzen processors, but Principled Technologies chose to enable the feature on the eight-core Ryzen 7 2700X anyway. That turns the processor into a four-core chip.

As you might expect, this effectively cripples the processor in heavily threaded gaming benchmarks, and the test suite included a few impacted titles, like Ashes of the Singularity. AMD's Ryzen Master software actually locks the Game Mode feature out on the Ryzen 2700X because the chip can't switch between 'local' and 'dynamic' modes, meaning the firm might have enabled the compatibility mode that accomplishes the same task of disabling four cores.

The firm also tested the games with a 1920 x 1080 resolution, which has drawn the ire of many enthusiasts because we wouldn't expect gamers with a pricey Core i9-9900K processor to play at that resolution. However, the fact remains that testing at lower resolutions is a solid test methodology if you're focusing specifically on the host processor, as reducing the graphics bottleneck as much as possible allows for true measurement of CPU performance, rather than a graphics-imposed bottleneck. In fact, the majority of review sites still test CPU gaming performance at this resolution.

We've reached out to Principled Technologies for comment and have lodged follow up questions with Intel. For now, in spite of Intel's statement, it appears that the tests are questionable at best. But that's the norm for all vendor-supplied benchmarks. As usual, we advise readers to wait for reviews instead of plunking down their hard-earned cash based solely on vendor benchmarks.

EDIT 10/9 3:20pm PST: Clarified memory and game mode settings.