Challenging FPS: Testing SLI And CrossFire Using Video Capture

Frames Per Second: Why The World Was Wrong

"You take the red pill - you stay in Wonderland, and I show you how deep the rabbit hole goes."
- Morpheus, The Matrix

Over the years, we've accumulated mountains of data from benchmarking tools like Fraps and metrics built-in to top titles to help us evaluate performance. Historically, that information gave us our impression of how much faster one graphics card is than another, or what speed-up could be expected from a second GPU in CrossFire or SLI.

As a rule, human beings don't respond well when their beliefs are challenged. But how would you feel if I told you that the frames-per-second method for conveying performance, as it's often presented, is fundamentally flawed? It's tough to accept, right? And, to be honest, that was my first reaction the first time I heard that Scott Wasson at The Tech Report was checking into frame times using Fraps. His initial look and continued persistence was largely responsible for drawing attention to performance "inside the second," which is often discussed in terms of uneven or stuttery playback, even in the face of high average frame rates.

I still remember talking about this with Scott roughly two years ago, and we're still left with more questions than answers, despite his impressive volume of work over that time. There are a couple of reasons the escalation of this issue has taken so long.

First, as mentioned, even open-minded enthusiasts are uncomfortable with fundamental changes to what they took for granted previously (after all, that means we, you, and much of the industry was often wrong with our analysis). Nobody wants to believe that the information we were gleaning previously wasn't necessarily precise. So, many folks shied away from it for as long as possible.

Second, and perhaps even more technically-correct, there is no complete replacement for reporting average frame rate. Frame times and latency are not perfect answers to the problem; there are other variables in play, including where Fraps pulls its information from the graphics pipeline. At the end of the day, there is no metric we can use to definitively compare the smoothness of video performance based exclusively on objective observation.

That's what we're looking for; that's the Holy Grail. We'd need something to replace FPS. The bad news is that we're not there yet.

But frames per second is far from a useless yardstick. It reliably tells us when a piece of hardware delivers poor performance. When you see a card averaging less than 15 FPS, for instance, you know that combination of settings isn't running fluidly enough for a perceived sense of realism. There is no ambiguity in that. Unfortunately, averaging frames per second does not help distinguish between the consistency of rendered frames, particularly when two solutions serve up high frame rates and would appear to perform comparably. 

It's not all doom and gloom, though. This is an exciting time to be involved in PC hardware, and graphics performance gives us a new frontier to explore. There are a lot of smart people working on this problem, and it's something that'll invariably be conquered. For our part, we've put our own research into the question of smoothness, which you've recently seen reflected as charts that include average frame rates, minimum frame rates, frame rates over time, and frame time variance. None of those address the challenge completely, but they help paint a more complete picture when it comes to choosing the right graphics card for your games.

Today, we're exploring another tool that's going to help us dig into the performance of graphics cards (particularly multi-GPU configurations): Nvidia's Frame Capture Analysis Tool, or FCAT.

  • kajunchicken
    Hopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...
  • cleeve
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...
    FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
  • cangelini
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.
  • cravin
    I wish there were an easy way to make my frame rates not dip and spike so much. A lot of times it can go up to like 120fps but then dipts down to 60 -70. It makes it look super choppy and ugly. I know I could limit it at 60 frames a second, but wouldn't that just be like vsync? Would that have input lag.
  • DarkMantle
    Good review, but honestly I wouldnt use a tool touched by Nvidia to test AMD hardware, Nvidia has a track record of crippling the competition's hardware every chance they have. Also, i was checking prices in Newegg and to be honest the HD7870 is much cheaper than the GTX660Ti, why didn't you use 7870LE (Tahiti core) for this test? The price is much more closer.
    The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though.
  • krneki_05
    Vsync would only cut out the frames above 60FPS, so you would still have the FPS drops. but 60 or 70 FPS is more then you need (unless you are using 120Hz monitor and you have super awesome eyes to see the difference between 60 and 120FPS, that some do). No, the choppy felling you have must be something else not the frames.
  • BS Marketing
    Does it really matter? Over 60 FPS there will be screen tearing. So why is this sudden fuss? I guess nvidia marketing engine is in full flow. Only explanation is that Nvidia is really scared now, trying everything in there power to deceive people.
  • rojodogg
    I enjoyed the artical and it was very informative, I look forward to more testing of other GPU-s in the future.
  • bystander
    Great article.

    But as great as the review is, I feel one thing that review sites have dropped the ball on is the lack of v-sync comparisons. A lot of people play with v-sync, and while a 60hz monitor is going to limit what you can test, you could get a 120hz or 144hz monitor and see how they behave with v-sync on.

    And the toughest thing of all, is how can microstutter be more accurately quantified. Not counting the runt frames gives a more accurate representation of FPS, but does not quantify microstutter that may be happening as a result.

    It seems the more info we get, the more questions I have.
  • rene13cross
    @DarkMantle, exactly my thinking. I don't want to sound like a paranoid goof who thinks everything is a conspiracy but a test suite created by Nvidia to test AMD hardware doesn't sound like a very trustworthy test. I'm not saying that the results here are all false but Nvidia has had a slight history in the past with attempting to present the competition in an unfair light.