It's a basic rule of benchmarking. When you're comparing two competing products, you need to make all the testing conditions -- especially the settings -- as similar as possible. Otherwise, you're just comparing apples to crab apples, which is what may have happened in a recent benchmark report that showed Intel's new Core i9-9900K outperforming AMD's Ryzen 7 2700X in several games.
As part of its Coffee Lake Refresh launch, Intel paid test firm Principled Technologies to benchmark its new processors and compare them to chips in AMD's current lineup (PDF). However, Hardware Unboxed spotted a few test conditions that obviously could skew the results in favor of Intel's processor, including using a less-capable CPU cooler on AMD's chip, overclocking the RAM on Intel's platform only, and disabling half the cores on an AMD Ryzen processor.
We reached out to Intel for comment and got the following reply:
"We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world’s best gaming processor. Principled Technologies conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third-party testing in the coming weeks."
As you can see, Intel is standing by its assertion that the Core i9-9900K is the "world's best gaming processor," and given its specifications and Intel's current dominance in absolute gaming performance, the statement is likely true. However, Intel doesn't go as far as a full-throated endorsement of the report, instead saying the results are "consistent with what we've seen in our labs," which is an important distinction.
We also reached out to Principled Technologies for comment but didn't receive a response as of press time.
|Game||Intel Core i9-9900K||AMD Ryzen 7 2700X|
|Total War: Warhammer II Battle (fps)||110.4||100.7|
|Counter-Strike Global Offensive (fps)||440||295.2|
|Gears of War 4 Overall (fps)||196.7||128|
|War Thunder Tank Battle (fps)||147.9||126.2|
|Ashes of the Singularity (fps)||57.3||38.3|
|Forza Motorsport 7 (fps)||203.6||150.9|
|Assassin's Creed Origins (benchmark score)||13,862||10,531|
|Far Cry 5 (fps)||136||104|
|World of Warcraft: Battle of Azeroth (fps)||126||99.2|
There are few absolute truths, but the old mantra of "never trust vendor's benchmarks" is certainly one of them. That means today's findings that the benchmarks could be flawed isn't entirely surprising. We've seen misleading reports in the past from almost every vendor, but it is concerning that this report comes during Intel's 10-day preorder window.
Here are some of the controversial results in Principled Technologies' report.
Principled Technologies' benchmarks pit the Core i9-9900K, Intel's new flagship eight-core processor for mainstream systems, against AMD's mainstream desktop Ryzen 7 2700X. The 19 game benchmarks in the report paint a very convincing picture of the Core i9-9900K's total dominance over the Ryzen 7 2700X in gaming, but a deeper look at the system settings revealed a few questionable, and one downright unbelievable, configuration that could skew the results in Intel's favor.
The missteps begin with system memory. Principled Technologies used the XMP profile, which automatically assigns tight timings for the Corsair Vengeance DDR4-3200 memory (running at DDR4-2666) installed in the Intel system. In contrast, the company set the Ryzen 7 2700X's memory configuration, which was fully populated with four double-sided DIMMs, at the stock DDR4-2933. This setup allows the system to assign timings for the memory, which often leads to suboptimal memory performance.
As we've proven time and again, AMD's Zen architecture is extremely sensitive to memory performance, particularly in games. As such, even small variations in settings result in improved gaming performance with Ryzen processors, meaning this mismatch could give Intel at least some advantage in the gaming tests.
Principled Technologies also used the stock Ryzen 7 2700X Wraith Prism cooler with the AMD system but installed a premium Noctua NH-U14S cooler on the Intel processor. Again, as we've proven in the past, improved cooling benefits both AMD and Intel's chips by allowing the processors to operate at their Boost frequencies more frequently, and then maintain the heightened clock speeds for longer periods of time.
Slight differences in cooling solutions can have an impact on the test setup. In this case, the Intel system could expend more waste heat than the AMD system, which would provide it with an advantage. Intel's new K-series processor doesn't come with a stock cooler, so the company had to use a third-party solution, but sound testing methodology dictates that coolers for both systems should be of equal quality. That didn't happen here.
Perhaps the most damning admission involves AMD's Game Mode feature that AMD created for its Threadripper processors. Game mode essentially disables half of the processors' available cores to circumvent the intricacies of the Threadripper architecture. Those same principles don't apply to the mainstream Ryzen processors, but Principled Technologies chose to enable the feature on the eight-core Ryzen 7 2700X anyway. That turns the processor into a four-core chip.
As you might expect, this effectively cripples the processor in heavily threaded gaming benchmarks, and the test suite included a few impacted titles, like Ashes of the Singularity. AMD's Ryzen Master software actually locks the Game Mode feature out on the Ryzen 2700X because the chip can't switch between 'local' and 'dynamic' modes, meaning the firm might have enabled the compatibility mode that accomplishes the same task of disabling four cores.
The firm also tested the games with a 1920 x 1080 resolution, which has drawn the ire of many enthusiasts because we wouldn't expect gamers with a pricey Core i9-9900K processor to play at that resolution. However, the fact remains that testing at lower resolutions is a solid test methodology if you're focusing specifically on the host processor, as reducing the graphics bottleneck as much as possible allows for true measurement of CPU performance, rather than a graphics-imposed bottleneck. In fact, the majority of review sites still test CPU gaming performance at this resolution.
We've reached out to Principled Technologies for comment and have lodged follow up questions with Intel. For now, in spite of Intel's statement, it appears that the tests are questionable at best. But that's the norm for all vendor-supplied benchmarks. As usual, we advise readers to wait for reviews instead of plunking down their hard-earned cash based solely on vendor benchmarks.
EDIT 10/9 3:20pm PST: Clarified memory and game mode settings.
Stay on the Cutting Edge
Join the experts who read Tom's Hardware for the inside track on enthusiast PC tech news — and have for over 25 years. We'll send breaking news and in-depth reviews of CPUs, GPUs, AI, maker hardware and more straight to your inbox.
Paul Alcorn is the Deputy Managing Editor for Tom's Hardware US. He writes news and reviews on CPUs, storage and enterprise hardware.
What were they using for a GPU? I get about 50 fps on heavy batches for Ashes with a 2700x and a GTX 1070 on high. And I think you meant dual channel memory for the 2700x.Reply
If you have to SAY that your company is "Principled", well, then, you might NOT be.Reply
It might even just mean your principles are for-hire.
Looking forward to Tomshardware's benchmark.Reply
HardwareUnboxed is just trying to get views. If you actaully looked at the full test report pdf, the Intel systems were actually tested at 2666MHz ram speed vs. 2933MHz for Ryzen. The Wraith Prism cooler is what comes with the CPU. Are you saying that the Wraith Prism cooler is no good? Cause everyone says that it's the best AMD cooler. Are you saying that they should put a crappy cooler on the Intel system to hold it back? And finally testing CPUs at 1080p is the standard. No one tests CPUs at 4K.Reply
This is the link to the test report.Reply
I think what everyone is saying is this this review seems bought and paid for so nobody trusts the numbers. I certainly do not trust these numbers, sure they may be correct, but I will wait for impartial reviews.Reply
I am glad TH is taking the high road on this versus Nvidia's RTX recommendations.
Ryzen 7 2700X does not have quad channel memory.Reply
And yes running game mode on it is ultra shady. These results are useless.
What was done at Principled Technologies was in a very scew manner first they didn't enable XMP profile , 2nd they intentional disabled 4 cores on the 2700x how is that a fairly equal test and then they upgrade the HSF used on the i9 but not do the same for the 2700x i wouldn't be surprised if they also held the fan speed to 50%.Reply
This was released at a time of a NDA on all reviewer's site's. So by the time any testing was allowed the word would have already sent out as truth with no one to disput their claims. Very DIRTY Tatics by INTEL here