Sign in with
Sign up | Sign in

Challenging FPS: Testing SLI And CrossFire Using Video Capture

Challenging FPS: Testing SLI And CrossFire Using Video Capture
By

What if the performance data you used for deciding which graphics card to buy was flawed? We're taking a deeper look at some of the problems with benchmarking multi-GPU configs using conventional tools. Nvidia's new FCAT suite helps us collect more info.

"You take the red pill - you stay in Wonderland, and I show you how deep the rabbit hole goes."
- Morpheus, The Matrix

Over the years, we've accumulated mountains of data from benchmarking tools like Fraps and metrics built-in to top titles to help us evaluate performance. Historically, that information gave us our impression of how much faster one graphics card is than another, or what speed-up could be expected from a second GPU in CrossFire or SLI.

As a rule, human beings don't respond well when their beliefs are challenged. But how would you feel if I told you that the frames-per-second method for conveying performance, as it's often presented, is fundamentally flawed? It's tough to accept, right? And, to be honest, that was my first reaction the first time I heard that Scott Wasson at The Tech Report was checking into frame times using Fraps. His initial look and continued persistence was largely responsible for drawing attention to performance "inside the second," which is often discussed in terms of uneven or stuttery playback, even in the face of high average frame rates.

I still remember talking about this with Scott roughly two years ago, and we're still left with more questions than answers, despite his impressive volume of work over that time. There are a couple of reasons the escalation of this issue has taken so long.

First, as mentioned, even open-minded enthusiasts are uncomfortable with fundamental changes to what they took for granted previously (after all, that means we, you, and much of the industry was often wrong with our analysis). Nobody wants to believe that the information we were gleaning previously wasn't necessarily precise. So, many folks shied away from it for as long as possible.

Second, and perhaps even more technically-correct, there is no complete replacement for reporting average frame rate. Frame times and latency are not perfect answers to the problem; there are other variables in play, including where Fraps pulls its information from the graphics pipeline. At the end of the day, there is no metric we can use to definitively compare the smoothness of video performance based exclusively on objective observation.

That's what we're looking for; that's the Holy Grail. We'd need something to replace FPS. The bad news is that we're not there yet.


But frames per second is far from a useless yardstick. It reliably tells us when a piece of hardware delivers poor performance. When you see a card averaging less than 15 FPS, for instance, you know that combination of settings isn't running fluidly enough for a perceived sense of realism. There is no ambiguity in that. Unfortunately, averaging frames per second does not help distinguish between the consistency of rendered frames, particularly when two solutions serve up high frame rates and would appear to perform comparably. 

It's not all doom and gloom, though. This is an exciting time to be involved in PC hardware, and graphics performance gives us a new frontier to explore. There are a lot of smart people working on this problem, and it's something that'll invariably be conquered. For our part, we've put our own research into the question of smoothness, which you've recently seen reflected as charts that include average frame rates, minimum frame rates, frame rates over time, and frame time variance. None of those address the challenge completely, but they help paint a more complete picture when it comes to choosing the right graphics card for your games.

Today, we're exploring another tool that's going to help us dig into the performance of graphics cards (particularly multi-GPU configurations): Nvidia's Frame Capture Analysis Tool, or FCAT.

Display all 199 comments.
This thread is closed for comments
Top Comments
  • 20 Hide
    DarkMantle , March 27, 2013 8:05 AM
    Good review, but honestly I wouldnt use a tool touched by Nvidia to test AMD hardware, Nvidia has a track record of crippling the competition's hardware every chance they have. Also, i was checking prices in Newegg and to be honest the HD7870 is much cheaper than the GTX660Ti, why didn't you use 7870LE (Tahiti core) for this test? The price is much more closer.
    The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though.
  • 20 Hide
    cleeve , March 27, 2013 7:42 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...


    FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
  • 15 Hide
    cangelini , March 27, 2013 7:45 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...

    And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.
Other Comments
  • 20 Hide
    cleeve , March 27, 2013 7:42 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...


    FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
  • 15 Hide
    cangelini , March 27, 2013 7:45 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...

    And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.
  • 20 Hide
    DarkMantle , March 27, 2013 8:05 AM
    Good review, but honestly I wouldnt use a tool touched by Nvidia to test AMD hardware, Nvidia has a track record of crippling the competition's hardware every chance they have. Also, i was checking prices in Newegg and to be honest the HD7870 is much cheaper than the GTX660Ti, why didn't you use 7870LE (Tahiti core) for this test? The price is much more closer.
    The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though.
  • 10 Hide
    krneki_05 , March 27, 2013 8:06 AM
    Vsync would only cut out the frames above 60FPS, so you would still have the FPS drops. but 60 or 70 FPS is more then you need (unless you are using 120Hz monitor and you have super awesome eyes to see the difference between 60 and 120FPS, that some do). No, the choppy felling you have must be something else not the frames.
  • 1 Hide
    rojodogg , March 27, 2013 8:20 AM
    I enjoyed the artical and it was very informative, I look forward to more testing of other GPU-s in the future.
  • 14 Hide
    bystander , March 27, 2013 8:22 AM
    Great article.

    But as great as the review is, I feel one thing that review sites have dropped the ball on is the lack of v-sync comparisons. A lot of people play with v-sync, and while a 60hz monitor is going to limit what you can test, you could get a 120hz or 144hz monitor and see how they behave with v-sync on.

    And the toughest thing of all, is how can microstutter be more accurately quantified. Not counting the runt frames gives a more accurate representation of FPS, but does not quantify microstutter that may be happening as a result.

    It seems the more info we get, the more questions I have.
  • 1 Hide
    rene13cross , March 27, 2013 8:28 AM
    @DarkMantle, exactly my thinking. I don't want to sound like a paranoid goof who thinks everything is a conspiracy but a test suite created by Nvidia to test AMD hardware doesn't sound like a very trustworthy test. I'm not saying that the results here are all false but Nvidia has had a slight history in the past with attempting to present the competition in an unfair light.
  • 4 Hide
    bystander , March 27, 2013 8:28 AM
    BigMack70It would also be interesting to see how much using a framerate limiter through something like Afterburner helps things... my experience is that framerate limiter + vsync = no (or almost no) perceivable stutter, even where it may have been really awful beforehand.That's a good deal of extra work and data to present, though...


    Yeah, that is a big part of why I'd like to see v-sync used in a review some time. It also removes tearing, and is the primary way I play; v-sync on a 120hz monitor.
  • 0 Hide
    Onus , March 27, 2013 8:29 AM
    This will take considerable time to digest, but my quick take on it so far is that I am very glad I have never wasted time or money on a [midrange] Crossfire setup. SLI looks a lot more viable, but nVidia is no less guilty than AMD of releasing the occasional bum drivers.
    Particularly after re-reading pp1-2, please clarify, runts [and drops] are not an issue in single-card setups?
  • 6 Hide
    bystander , March 27, 2013 8:32 AM
    rene13cross@DarkMantle, exactly my thinking. I don't want to sound like a paranoid goof who thinks everything is a conspiracy but a test suite created by Nvidia to test AMD hardware doesn't sound like a very trustworthy test. I'm not saying that the results here are all false but Nvidia has had a slight history in the past with attempting to present the competition in an unfair light.

    The test is one that AMD wanted as well. Well, at least that is what they are saying now, because it tests the output, not the start of the rendering process. I'm not sure how this type of test could skew results, as it just takes the frames like a monitor does, and shows us what the monitor shows.

    The part you could possibly quibble over is what quantifies a runt frame.
  • 2 Hide
    ojas , March 27, 2013 8:35 AM
    Don/Chris:

    There was a interesting AMD-backed story the other day on AT (not that it matters too much, both AMD and Nvidia seem to not like FRAPS that much), AMD apparently uses stuff like GPUView from MS.
    http://www.anandtech.com/show/6857/amd-stuttering-issues-driver-roadmap-fraps

    Intel's also got something called Graphics Performance Analyzers, which seems to be similar to GPUView.
    http://software.intel.com/en-us/vcsource/tools/intel-gpa
  • 2 Hide
    ingtar33 , March 27, 2013 8:36 AM
    love the article, though it mostly measures the unmeasurable in ways the eye can't tell the difference. I appreciate the reviewer's honesty when he admitted only 2 titles where he could tell the poorer performing card in the test were performing poorer... this tells me the settings for the test... or what the test is measuring is largely undetectable and of questionable value.

    What needs to be done with FPS/FRAPs/whatever is a practical tested and verifiable standard needs to be created which accurately portrays the playable experience. sorta a meta rating which incorporates all these sub criteria into a number... which will let us know how silky smooth the play experience will be with a gaming title.

    of course, there is the added issue with an nvidia program being used to measure an AMD part... with the way intel used to (or might still according to some people) influence certain benching programs it's beyond problematic, especially with the way NVidia has played in the past with certain competition, to use a software program made by one of the competitors. if their methodology has value, it should be re-engineered to insure impartiality, and to prevent the obvious and expected fanboy mistrust.

    That said I agree with the author's general point... this is an exciting time to be an enthusiasts.
  • 4 Hide
    nukemaster , March 27, 2013 8:38 AM
    I too would like to see vsync comparisons.
  • 1 Hide
    bystander , March 27, 2013 8:38 AM
    krneki_05Vsync would only cut out the frames above 60FPS, so you would still have the FPS drops. but 60 or 70 FPS is more then you need (unless you are using 120Hz monitor and you have super awesome eyes to see the difference between 60 and 120FPS, that some do). No, the choppy felling you have must be something else not the frames.

    I read an article on Techreport recently that explained another big component to choppy game play is time syncing. One that no review site has ever tried to tackle. It is one thing to have evenly spaced frames, but what if those frames are not synced to the action? That would have more unsettling results.
  • 5 Hide
    ojas , March 27, 2013 8:42 AM
    Um, i'm still reading Batman:AC, two things:
    1. No FRAPS for Nvidia? How do we know FRAPS isn't causing an issue there?
    2. The Minimum FPS for the FRAPS measurement is actually lower than the hardware and practical. What's going on there, if FRAPS counts present() calls, then shouldn't it be more than the hardware FPS at the very least (unless i'm missing something, i think it should be the same at least).
  • 0 Hide
    ubercake , March 27, 2013 8:47 AM
    BS MarketingDoes it really matter? Over 60 FPS there will be screen tearing. So why is this sudden fuss? I guess nvidia marketing engine is in full flow. Only explanation is that Nvidia is really scared now, trying everything in there power to deceive people.

    No tearing on 120Hz monitors until you get over 120fps and even then tearing is no longer perceivable until you hit the mid 400s.

    Also, that is not the point of the article.

    This is a great article. It's consistent with others I've read on the subject. It is consistent to what is being published regarding information AMD is also supporting.

    I look forward to seeing what you do with the tweaks of the FCAT software to further define what equates to a "runt" frame. Seems like that could make an even greater difference. Defining a runt frame seems somewhat subjective. Seems like many more than 21 scan lines or less could define a runt and would seem dependent on the resolution somewhat?
  • 0 Hide
    blazorthon , March 27, 2013 8:50 AM
    BS MarketingDoes it really matter? Over 60 FPS there will be screen tearing. So why is this sudden fuss? I guess nvidia marketing engine is in full flow. Only explanation is that Nvidia is really scared now, trying everything in there power to deceive people.


    You can get screen tearing regardless of what FPS you have. You might get it even at 40FPS and you might not get it at 200FPS. Just because you're over 60FPS doesn't necessarily mean that you'll have screen tearing just as being under 60FPS doesn't mean that you won't get screen tearing.
Display more comments