What if the performance data you used for deciding which graphics card to buy was flawed? We're taking a deeper look at some of the problems with benchmarking multi-GPU configs using conventional tools. Nvidia's new FCAT suite helps us collect more info.
"You take the red pill - you stay in Wonderland, and I show you how deep the rabbit hole goes."
- Morpheus, The Matrix
Over the years, we've accumulated mountains of data from benchmarking tools like Fraps and metrics built-in to top titles to help us evaluate performance. Historically, that information gave us our impression of how much faster one graphics card is than another, or what speed-up could be expected from a second GPU in CrossFire or SLI.
As a rule, human beings don't respond well when their beliefs are challenged. But how would you feel if I told you that the frames-per-second method for conveying performance, as it's often presented, is fundamentally flawed? It's tough to accept, right? And, to be honest, that was my first reaction the first time I heard that Scott Wasson at The Tech Report was checking into frame times using Fraps. His initial look and continued persistence was largely responsible for drawing attention to performance "inside the second," which is often discussed in terms of uneven or stuttery playback, even in the face of high average frame rates.
I still remember talking about this with Scott roughly two years ago, and we're still left with more questions than answers, despite his impressive volume of work over that time. There are a couple of reasons the escalation of this issue has taken so long.
First, as mentioned, even open-minded enthusiasts are uncomfortable with fundamental changes to what they took for granted previously (after all, that means we, you, and much of the industry was often wrong with our analysis). Nobody wants to believe that the information we were gleaning previously wasn't necessarily precise. So, many folks shied away from it for as long as possible.
Second, and perhaps even more technically-correct, there is no complete replacement for reporting average frame rate. Frame times and latency are not perfect answers to the problem; there are other variables in play, including where Fraps pulls its information from the graphics pipeline. At the end of the day, there is no metric we can use to definitively compare the smoothness of video performance based exclusively on objective observation.
That's what we're looking for; that's the Holy Grail. We'd need something to replace FPS. The bad news is that we're not there yet.

But frames per second is far from a useless yardstick. It reliably tells us when a piece of hardware delivers poor performance. When you see a card averaging less than 15 FPS, for instance, you know that combination of settings isn't running fluidly enough for a perceived sense of realism. There is no ambiguity in that. Unfortunately, averaging frames per second does not help distinguish between the consistency of rendered frames, particularly when two solutions serve up high frame rates and would appear to perform comparably.
It's not all doom and gloom, though. This is an exciting time to be involved in PC hardware, and graphics performance gives us a new frontier to explore. There are a lot of smart people working on this problem, and it's something that'll invariably be conquered. For our part, we've put our own research into the question of smoothness, which you've recently seen reflected as charts that include average frame rates, minimum frame rates, frame rates over time, and frame time variance. None of those address the challenge completely, but they help paint a more complete picture when it comes to choosing the right graphics card for your games.
Today, we're exploring another tool that's going to help us dig into the performance of graphics cards (particularly multi-GPU configurations): Nvidia's Frame Capture Analysis Tool, or FCAT.
- Frames Per Second: Why The World Was Wrong
- Multi-Card Graphics Problems, And A Solution: Nvidia's FCAT
- Test System And Benchmarks
- Results: Batman Arkham City
- Results: Borderlands 2
- Results: Crysis 3
- Results: F1 2012
- Results: Far Cry 3
- Results: Hitman Absolution
- Results: Metro 2033
- Results: Elder Scrolls V: Skyrim
- Results: Tomb Raider
- When Frame Rates Aren't What They Seem...
The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though.
FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.
FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.
The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though.
But as great as the review is, I feel one thing that review sites have dropped the ball on is the lack of v-sync comparisons. A lot of people play with v-sync, and while a 60hz monitor is going to limit what you can test, you could get a 120hz or 144hz monitor and see how they behave with v-sync on.
And the toughest thing of all, is how can microstutter be more accurately quantified. Not counting the runt frames gives a more accurate representation of FPS, but does not quantify microstutter that may be happening as a result.
It seems the more info we get, the more questions I have.
Yeah, that is a big part of why I'd like to see v-sync used in a review some time. It also removes tearing, and is the primary way I play; v-sync on a 120hz monitor.
Particularly after re-reading pp1-2, please clarify, runts [and drops] are not an issue in single-card setups?
The test is one that AMD wanted as well. Well, at least that is what they are saying now, because it tests the output, not the start of the rendering process. I'm not sure how this type of test could skew results, as it just takes the frames like a monitor does, and shows us what the monitor shows.
The part you could possibly quibble over is what quantifies a runt frame.
There was a interesting AMD-backed story the other day on AT (not that it matters too much, both AMD and Nvidia seem to not like FRAPS that much), AMD apparently uses stuff like GPUView from MS.
http://www.anandtech.com/show/6857/amd-stuttering-issues-driver-roadmap-fraps
Intel's also got something called Graphics Performance Analyzers, which seems to be similar to GPUView.
http://software.intel.com/en-us/vcsource/tools/intel-gpa
What needs to be done with FPS/FRAPs/whatever is a practical tested and verifiable standard needs to be created which accurately portrays the playable experience. sorta a meta rating which incorporates all these sub criteria into a number... which will let us know how silky smooth the play experience will be with a gaming title.
of course, there is the added issue with an nvidia program being used to measure an AMD part... with the way intel used to (or might still according to some people) influence certain benching programs it's beyond problematic, especially with the way NVidia has played in the past with certain competition, to use a software program made by one of the competitors. if their methodology has value, it should be re-engineered to insure impartiality, and to prevent the obvious and expected fanboy mistrust.
That said I agree with the author's general point... this is an exciting time to be an enthusiasts.
I read an article on Techreport recently that explained another big component to choppy game play is time syncing. One that no review site has ever tried to tackle. It is one thing to have evenly spaced frames, but what if those frames are not synced to the action? That would have more unsettling results.
1. No FRAPS for Nvidia? How do we know FRAPS isn't causing an issue there?
2. The Minimum FPS for the FRAPS measurement is actually lower than the hardware and practical. What's going on there, if FRAPS counts present() calls, then shouldn't it be more than the hardware FPS at the very least (unless i'm missing something, i think it should be the same at least).
No tearing on 120Hz monitors until you get over 120fps and even then tearing is no longer perceivable until you hit the mid 400s.
Also, that is not the point of the article.
This is a great article. It's consistent with others I've read on the subject. It is consistent to what is being published regarding information AMD is also supporting.
I look forward to seeing what you do with the tweaks of the FCAT software to further define what equates to a "runt" frame. Seems like that could make an even greater difference. Defining a runt frame seems somewhat subjective. Seems like many more than 21 scan lines or less could define a runt and would seem dependent on the resolution somewhat?
You can get screen tearing regardless of what FPS you have. You might get it even at 40FPS and you might not get it at 200FPS. Just because you're over 60FPS doesn't necessarily mean that you'll have screen tearing just as being under 60FPS doesn't mean that you won't get screen tearing.