Sign in with
Sign up | Sign in

When Frame Rates Aren't What They Seem...

Challenging FPS: Testing SLI And CrossFire Using Video Capture
By

We've known about the existence of Nvidia's frame metering technology for years, and its efforts to quantify the benefits of spacing frames out evenly, minimizing dropped and runt frames, for months. It was only recently, however, that the company was willing or able to show off the fruits of its development efforts. Even today, the tools can be a little finicky. We would have had even more performance data, even, except that our X79 Express-based benchmark platform was spitting out FCAT data that clearly wasn't right. Switching over to Z77 Express at the last minute gave us the results we were looking for.

As we expected, both from The Tech Report's background with this and Nvidia's in-house examples, we see that AMD's Radeon HD 7870s in CrossFire tend to suffer more dropped and runt frames than a pair of GeForce GTX 660 Tis in SLI. This addresses much of the trepidation about multi-card configurations expressed in Best Graphics Cards for the Money, confirming that two of Nvidia's boards appear more appealing. AMD even admits it's playing catch-up in this area, and is addressing its shortcomings through successive driver improvements.

The Datapath card we used for video captureThe Datapath card we used for video capture

In any case, it's telling that Nvidia put in all of this effort to quantifying graphics performance in a scientifically sound way. The company deserves credit for putting resources into something we wouldn't have been able to develop on our own, even if its motivation wasn't altruistic. The fact is that FCAT gives us a tool for evaluating something we couldn't accurately measure before. We now have two Tom's Hardware labs enabled with the hardware and software to run tests using FCAT, and we're already in the process of testing for a much more comprehensive Part 2.

Having said all of that, we're once again left with questions, even as we uncover a number of answers. While it seems obvious that a runt frame of fewer than 21 scan lines contributes little (or nothing) to the smoothness of a game, would a hardcore gamer see a quality improvement if we split the screen evenly by two, three, or even 50 frames composed of 22 or more lines on the screen? The FCAT tool is built to facilitate user-specified definitions of how large a runt frame can be, and we'll need to play with the script's switches to really dial-in our own recipe for performance evaluation. What we're presenting today is really what FCAT can do out of its proverbial box.

But questions like that tell us we have a lot more work to do. Hopefully, innovative tools like Nvidia's FCAT help us solve them. Keep an eye on this space for our upcoming follow-up.

Ask a Category Expert

Create a new thread in the Reviews comments forum about this subject

Example: Notebook, Android, SSD hard drive

Display all 199 comments.
This thread is closed for comments
Top Comments
  • 20 Hide
    DarkMantle , March 27, 2013 8:05 AM
    Good review, but honestly I wouldnt use a tool touched by Nvidia to test AMD hardware, Nvidia has a track record of crippling the competition's hardware every chance they have. Also, i was checking prices in Newegg and to be honest the HD7870 is much cheaper than the GTX660Ti, why didn't you use 7870LE (Tahiti core) for this test? The price is much more closer.
    The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though.
  • 20 Hide
    cleeve , March 27, 2013 7:42 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...


    FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
  • 15 Hide
    cangelini , March 27, 2013 7:45 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...

    And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.
Other Comments
  • 20 Hide
    cleeve , March 27, 2013 7:42 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...


    FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
  • 15 Hide
    cangelini , March 27, 2013 7:45 AM
    kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...

    And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.
  • 20 Hide
    DarkMantle , March 27, 2013 8:05 AM
    Good review, but honestly I wouldnt use a tool touched by Nvidia to test AMD hardware, Nvidia has a track record of crippling the competition's hardware every chance they have. Also, i was checking prices in Newegg and to be honest the HD7870 is much cheaper than the GTX660Ti, why didn't you use 7870LE (Tahiti core) for this test? The price is much more closer.
    The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though.
  • 10 Hide
    krneki_05 , March 27, 2013 8:06 AM
    Vsync would only cut out the frames above 60FPS, so you would still have the FPS drops. but 60 or 70 FPS is more then you need (unless you are using 120Hz monitor and you have super awesome eyes to see the difference between 60 and 120FPS, that some do). No, the choppy felling you have must be something else not the frames.
  • 1 Hide
    rojodogg , March 27, 2013 8:20 AM
    I enjoyed the artical and it was very informative, I look forward to more testing of other GPU-s in the future.
  • 14 Hide
    bystander , March 27, 2013 8:22 AM
    Great article.

    But as great as the review is, I feel one thing that review sites have dropped the ball on is the lack of v-sync comparisons. A lot of people play with v-sync, and while a 60hz monitor is going to limit what you can test, you could get a 120hz or 144hz monitor and see how they behave with v-sync on.

    And the toughest thing of all, is how can microstutter be more accurately quantified. Not counting the runt frames gives a more accurate representation of FPS, but does not quantify microstutter that may be happening as a result.

    It seems the more info we get, the more questions I have.
  • 1 Hide
    rene13cross , March 27, 2013 8:28 AM
    @DarkMantle, exactly my thinking. I don't want to sound like a paranoid goof who thinks everything is a conspiracy but a test suite created by Nvidia to test AMD hardware doesn't sound like a very trustworthy test. I'm not saying that the results here are all false but Nvidia has had a slight history in the past with attempting to present the competition in an unfair light.
  • 4 Hide
    bystander , March 27, 2013 8:28 AM
    BigMack70It would also be interesting to see how much using a framerate limiter through something like Afterburner helps things... my experience is that framerate limiter + vsync = no (or almost no) perceivable stutter, even where it may have been really awful beforehand.That's a good deal of extra work and data to present, though...


    Yeah, that is a big part of why I'd like to see v-sync used in a review some time. It also removes tearing, and is the primary way I play; v-sync on a 120hz monitor.
  • 0 Hide
    Onus , March 27, 2013 8:29 AM
    This will take considerable time to digest, but my quick take on it so far is that I am very glad I have never wasted time or money on a [midrange] Crossfire setup. SLI looks a lot more viable, but nVidia is no less guilty than AMD of releasing the occasional bum drivers.
    Particularly after re-reading pp1-2, please clarify, runts [and drops] are not an issue in single-card setups?
  • 6 Hide
    bystander , March 27, 2013 8:32 AM
    rene13cross@DarkMantle, exactly my thinking. I don't want to sound like a paranoid goof who thinks everything is a conspiracy but a test suite created by Nvidia to test AMD hardware doesn't sound like a very trustworthy test. I'm not saying that the results here are all false but Nvidia has had a slight history in the past with attempting to present the competition in an unfair light.

    The test is one that AMD wanted as well. Well, at least that is what they are saying now, because it tests the output, not the start of the rendering process. I'm not sure how this type of test could skew results, as it just takes the frames like a monitor does, and shows us what the monitor shows.

    The part you could possibly quibble over is what quantifies a runt frame.
  • 2 Hide
    ojas , March 27, 2013 8:35 AM
    Don/Chris:

    There was a interesting AMD-backed story the other day on AT (not that it matters too much, both AMD and Nvidia seem to not like FRAPS that much), AMD apparently uses stuff like GPUView from MS.
    http://www.anandtech.com/show/6857/amd-stuttering-issues-driver-roadmap-fraps

    Intel's also got something called Graphics Performance Analyzers, which seems to be similar to GPUView.
    http://software.intel.com/en-us/vcsource/tools/intel-gpa
  • 2 Hide
    ingtar33 , March 27, 2013 8:36 AM
    love the article, though it mostly measures the unmeasurable in ways the eye can't tell the difference. I appreciate the reviewer's honesty when he admitted only 2 titles where he could tell the poorer performing card in the test were performing poorer... this tells me the settings for the test... or what the test is measuring is largely undetectable and of questionable value.

    What needs to be done with FPS/FRAPs/whatever is a practical tested and verifiable standard needs to be created which accurately portrays the playable experience. sorta a meta rating which incorporates all these sub criteria into a number... which will let us know how silky smooth the play experience will be with a gaming title.

    of course, there is the added issue with an nvidia program being used to measure an AMD part... with the way intel used to (or might still according to some people) influence certain benching programs it's beyond problematic, especially with the way NVidia has played in the past with certain competition, to use a software program made by one of the competitors. if their methodology has value, it should be re-engineered to insure impartiality, and to prevent the obvious and expected fanboy mistrust.

    That said I agree with the author's general point... this is an exciting time to be an enthusiasts.
  • 4 Hide
    nukemaster , March 27, 2013 8:38 AM
    I too would like to see vsync comparisons.
  • 1 Hide
    bystander , March 27, 2013 8:38 AM
    krneki_05Vsync would only cut out the frames above 60FPS, so you would still have the FPS drops. but 60 or 70 FPS is more then you need (unless you are using 120Hz monitor and you have super awesome eyes to see the difference between 60 and 120FPS, that some do). No, the choppy felling you have must be something else not the frames.

    I read an article on Techreport recently that explained another big component to choppy game play is time syncing. One that no review site has ever tried to tackle. It is one thing to have evenly spaced frames, but what if those frames are not synced to the action? That would have more unsettling results.
  • 5 Hide
    ojas , March 27, 2013 8:42 AM
    Um, i'm still reading Batman:AC, two things:
    1. No FRAPS for Nvidia? How do we know FRAPS isn't causing an issue there?
    2. The Minimum FPS for the FRAPS measurement is actually lower than the hardware and practical. What's going on there, if FRAPS counts present() calls, then shouldn't it be more than the hardware FPS at the very least (unless i'm missing something, i think it should be the same at least).
  • 0 Hide
    ubercake , March 27, 2013 8:47 AM
    BS MarketingDoes it really matter? Over 60 FPS there will be screen tearing. So why is this sudden fuss? I guess nvidia marketing engine is in full flow. Only explanation is that Nvidia is really scared now, trying everything in there power to deceive people.

    No tearing on 120Hz monitors until you get over 120fps and even then tearing is no longer perceivable until you hit the mid 400s.

    Also, that is not the point of the article.

    This is a great article. It's consistent with others I've read on the subject. It is consistent to what is being published regarding information AMD is also supporting.

    I look forward to seeing what you do with the tweaks of the FCAT software to further define what equates to a "runt" frame. Seems like that could make an even greater difference. Defining a runt frame seems somewhat subjective. Seems like many more than 21 scan lines or less could define a runt and would seem dependent on the resolution somewhat?
  • 0 Hide
    blazorthon , March 27, 2013 8:50 AM
    BS MarketingDoes it really matter? Over 60 FPS there will be screen tearing. So why is this sudden fuss? I guess nvidia marketing engine is in full flow. Only explanation is that Nvidia is really scared now, trying everything in there power to deceive people.


    You can get screen tearing regardless of what FPS you have. You might get it even at 40FPS and you might not get it at 200FPS. Just because you're over 60FPS doesn't necessarily mean that you'll have screen tearing just as being under 60FPS doesn't mean that you won't get screen tearing.
Display more comments