Introducing Our Reference System And Methodology For 2014
Our Mission: The Best Data Possible
Over the past year or so, you've seen us make some significant changes to many of our graphics card reviews. For example, whenever possible, we employ the Frame Capture Analysis Tool, dubbed FCAT and covered in depth in Challenging FPS: Testing SLI And CrossFire Using Video Capture. Our performance results are consequently more accurate than they've ever been.
Recently, we decided to dig deeper into power consumption as well, approaching it with the same precision, and you've likely seen the product of that in our more recent launch coverage. We don't want to just give you a rough estimate of what a given graphics card draws. No, the goal here is to set a new standard for power measurement. This is particularly important to us at a time when GPU vendors aren't just talking raw performance any more. They're putting an emphasis on efficiency, valuing cards able to achieve high frame rates without dissipating a ton of heat.
A new methodology, developed in concert with an industry partner, allows us to ask (and answer) questions that couldn't be easily addressed in the past. Just days ago, in Radeon R9 295X2 8 GB Review: Project Hydra Gets Liquid Cooling, we were able to compare the Radeon R9 295X2, HD 7990, and HD 6990, telling you how much power each card pulled over its PCI Express slot and eight-pin connectors. In the days to come, we'll publish a follow-up explaining why all of this matters so much, allowing you to interpret the outcome of our testing more easily.
The other purpose of today's story is to introduce our 2014 Graphics Card Charts. To account for advances in display technology, we're testing synthetics and real-world games at two resolutions now: FHD (1920x1080) and UHD (3840x2160). You'll see a second set of resolutions and results a little later; those will account for entry-level boards, APUs, and Intel's integrated graphics solutions.
Benchmarking On A New Reference System
Over the course of the last year, we've seen plenty of cases where an Intel Core i7-3770K, even one overclocked to 4.5 GHz, can become a bottleneck. Fortunately, at the resolutions we picked and the settings we're running at, limitations should be few. To further assure this, we also built up a new reference machine based on Intel's Core i7-4930K operating at 4 GHz, a very fast quad-channel memory kit, and Asus' Rampage IV Black Edition motherboard.
Freezing the Current State to Ensure Fair Comparisons
The evolution of gaming confounds us every year. It's easy enough to adopt new titles and retire old ones in our reviews. But it's a lot harder to collect a ton of data for these persistent charts and watch it age over the course of 12 months. We know it'd be impossible to use every single popular game for this project, particularly since we're doing multiple benchmark runs for each one. So, our team picked 10 AAA titles with fairly long sequences and available settings to test. The suite we ended up with purposely strikes a balance between graphics card vendors, too.
We’re using both modern and older games, which accordingly present a range of challenges for the graphics hardware. We deactivated automatic updates, effectively freezing their current state as of early 2014. And the copy of Windows we're using is also kept from phoning home for patches. This configuration is saved as an image, allowing us to reuse it over and over. Whenever it's necessary, we also use new drivers. Sometimes that necessitates applying them retroactively and re-testing cards, particularly when the performance of a card is affected in a big way.
Current page: Introducing Our Reference System And Methodology For 2014Next Page The Components In Our Reference Build
Stay on the Cutting Edge
Join the experts who read Tom's Hardware for the inside track on enthusiast PC tech news — and have for over 25 years. We'll send breaking news and in-depth reviews of CPUs, GPUs, AI, maker hardware and more straight to your inbox.
Thank you Tom's team for updating the charts. You're my goto when I'm upgrading my rigs. I'll be waiting... Bring on yesterday's gems.Reply
Nice writeup; I look forward to seeing the new charts!Reply
First thing Tom need is to bench how PCIE 2.0 8x vs 16x perform on a modern top end GPU. Since 290X are passing the bandwidth from crossfire bridge to PCIE, may be is time to check them again? As I recall AMD do not recommend putting 290x XDMA crossfire on PCIE 2.0 8x. Please check this outReply
First it's great to see new charts.Reply
I was never a fan of this style of benchmarking. It sure gives clean graph of gpu capabilities which we always needed. I would love to see new bottleneck analysis. Or at least parallel test done on midrange PC.
Everyone should keep mind that these charts represent performance of <1% PC builds out there.
13278215 said:First thing Tom need is to bench how PCIE 2.0 8x vs 16x perform on a modern top end GPU. Since 290X are passing the bandwidth from crossfire bridge to PCIE, may be is time to check them again? As I recall AMD do not recommend putting 290x XDMA crossfire on PCIE 2.0 8x. Please check this out
If I recall correctly we are at this moment at the edge of PCI 2.0 x8 which = PCI 1.0 x16 . Next or following gen will finally outdate PCI 1.0 in single and PCI 2.0 in dual GPU configs as there will finally be noticeable bottle necks.
Any Steam OS or GNU/Linux benchmarks?Reply
It would be nice to add any opengl crossplattform game as any ioquake based one or something more modern and test it under MS WOS and under GNU / Linux
Better if it is future Steam OS to let us know the performance at the same game under MS WOS and under GNU/Linux.
Also it would be nice to test at MS WOS with and without antivirus, perhaps avast that is free or any other of your preference.
Last but not least, in opengl or in directx there are version changes and being able to split cards generations by opengl / directx version support would help as a current price / performance index based in your sponsored links prices.
No 720p tests?Reply
720p ( 1280x720 píxels = 921.600 píxels) is half 1080p more or less
1080p (1920×1080 píxels = 2.073.600 pixels)
And when a game is very demanding or you prefer to play with better graphics playing at 720p is a great option
Of course,latest best GPUs would be able to play at 4k and full graphics, but when we read the benchmarks we want to know also if our actual card CAN play at 720p (1k) or what the best ones can do at 1k to be able to compare
Also even it is not a standard or accurate, for benchmarking purposes calling 720p (1k) 1080p (2k) and 2160p (4K) wouldbeeasier to understand in a fast sight than UHD FHD and HDR, that can be used too UHD (4k) FHD (2k) HDR (1k)
720p does not stress most reasonably decent GPUs much and how many people would drop resolution to 720p these days with all the re-scaling artifacts that might add? In most cases, it would make more sense to stick with native resolution and tweak some of the more GPU/memory-intensive settings down a notch or two - at least I know I greatly prefer cleaner images over "details" that get blurred by the lower resolution and re-scaling that further distorts it.13278758 said:No 720p tests?
Considering how you can get 1080p displays for $100, I would call standardizing the GPU chart on 1080p fair enough: the people who can only afford a $100 display won't care much about enabling every bell and whistle and the people who want to max everything out likely won't be playing on $100 displays and $100 GPUs either.
I really like to see the charts on how much noise a video card's cooling fans make. That makes more of a difference to me as limiting something distracting that I hear every time I game versus getting a louder card with 10 fps more.Reply
I also like seeing how current cards stack up performance-wise to previous generations. That really helps when you're deciding whether to upgrade or not.
Thank you! Good reference article!Reply