Sign in with
Sign up | Sign in

Introducing A New Test: Frame Time Variance

Gaming Shoot-Out: 18 CPUs And APUs Under $200, Benchmarked
By

As you may have noticed in a handful of our more recent graphics stories, we're working toward a new type of test standard that measures the impact of changes in frame rate latency. Historically, average frame rate, represented by frames per second (FPS), was our primary go-to for comparing the performance of graphics cards relative to each other. However, Scott Wasson of The Tech Report has put in a tremendous effort demonstrating where average frame rate comes up short in characterizing your experience gaming on a specific graphics subsystem.

By now, whether through Scott's work or our own, you're probably familiar with the phenomenon known as micro-stuttering, which is often associated with multi-card CrossFire or SLI configurations. This occurs when the amount of time that passes between frames appearing on-screen is inconsistent, resulting in what appears to be choppy gameplay, even in spite of high average frame rates. For example, two different PCs, both generating an average of 24 frames per second, may convey different experiences, one stuttery and one smooth, if the amount of time between each frame is less regular on one and more regular on the other.

In the chart below, System A sees a consistent amount of time between those 24 frames, while System B does not. Therefore, you may notice a stuttering effect on System B, even though both machines still average 24 FPS.

As you can see, System B has four frames that take significantly longer to render. System A is more consistent. The issue is easy to identify when we go beyond the average frame per second rate, and look at the frame times inside that second. But because most of our time-based benchmarks run for a while (at least a minute) in order to generate plenty of data, that would give us 3,600 data points at 60 FPS, at least. That's simply too much to squeeze onto an easily-digestible chart. Zooming in to a portion of the graph helps. However, how do you pick the most relevant slice to dissect? There's no easy answer.

In addition, raw frame times aren't the end-all in performance analysis because high frame rates have low corresponding frame times and low frame rates have high frame times. What we're trying to find is the variance, the amount of time that anomalous frames stray from the ideal norm.

Our preference is to take this data and put it into a simple, meaningful format that's easy to understand and analyze. To do this, we won't scrutinize the individual frame times, but we'll look closely at the difference (or variance) between the time it takes to display a frame compared to the ideal time it should take to display the frame based on the average of the frames surrounding it.

For example, in the chart above the average frame time for System B is just under 40 milliseconds. But four frames from System B suffer from abnormally long lag times from about 10 to 20 milliseconds, compared to the 40 millisecond average.

To describe this phenomenon, the frame time variance chart we're introducing today includes the average time variance across the entire benchmark, the 75th percentile time variance, and the 95th percentile time variance. Percentiles show us how bad things get, on average, over a larger sample. As a case in point, the 75th percentile result shows us the longest (worst) frame time variance that we see in the shortest (best) 75 percent of the samples, and so on with the 95th percentile.

Below, you'll find an example of how our frame time variance chart would describe the difference between System A and System B in the consecutive frame time chart presented above.

As you can see, this chart does not reflect raw frame rates. That's not its job. And that's fine with us because we're still going to continue capturing average frame rate for the foreseeable future. It may not tell the entire performance story, but it remains an important metric. We're simply adding the new data to help fill in the blanks.

Our hope is that, by comparing the results across different CPUs, we'll be able to identify issues where some models experience significantly higher latencies than we previously quantified. As you'll see in the results, each game has a different average frame time variance, too.

Display all 278 comments.
This thread is closed for comments
Top Comments
  • 33 Hide
    ingtar33 , February 12, 2013 3:29 AM
    so... the amd chips test as good as the intel chips (sometimes better) in your latency test, yet your conclusion is yet again based on average FPS?

    what is the point of running the latency tests if you're not going to use it in your conclusion?
  • 29 Hide
    cleeve , February 12, 2013 5:13 AM
    ingtar33so... the amd chips test as good as the intel chips (sometimes better) in your latency test, yet your conclusion is yet again based on average FPS?what is the point of running the latency tests if you're not going to use it in your conclusion?


    We absolutely did take latency into account in our conclusion.
    I think the problem is that you totally misunderstand the point of measuring latency, and the impact of the results. Please read page 2, and the commentary next to the charts.

    To summarize, latency is only relevant if it's significant enough to notice. If it's not significant (and really, it wasn't in any of the tests we took except maybe in some dual-core examples), then, obviously, the frame rate is the relevant measurement.

    *IF* the latency *WAS* horrible, say, with a high-FPS CPU, then in that case latency would be taken into account in the recommendations. But the latencies were very small, and so they don't really factor in much. Any CPUs that could handle at least four threads did great, the latencies are so imperceptible that they don't matter.
  • 25 Hide
    e56imfg , February 12, 2013 3:21 AM
    Nice round up. It's nice to finally shine some light onto the new FX chips.
Other Comments
  • 25 Hide
    e56imfg , February 12, 2013 3:21 AM
    Nice round up. It's nice to finally shine some light onto the new FX chips.
  • -3 Hide
    esrever , February 12, 2013 3:27 AM
    Wow. Frame latencies are completely different than the results on the tech report. Weird.
  • 33 Hide
    ingtar33 , February 12, 2013 3:29 AM
    so... the amd chips test as good as the intel chips (sometimes better) in your latency test, yet your conclusion is yet again based on average FPS?

    what is the point of running the latency tests if you're not going to use it in your conclusion?
  • 18 Hide
    shikamaru31789 , February 12, 2013 3:37 AM
    I was hanging around on the site hoping this would finally get posted today. Looks like I got lucky. I'm definitely happy that newer titles are using more threads, which finally puts AMD back in the running in the budget range at least. Even APU's look like a better buy now, I can't wait to see some Richland and Kaveri APU tests. If one of them has a built in 7750 you could have a nice budget system, especially if you paired it with a discrete GPU for Crossfire.
  • 9 Hide
    hero1 , February 12, 2013 3:39 AM
    ingtar33so... the amd chips test as good as the intel chips (sometimes better) in your latency test, yet your conclusion is yet again based on average FPS?what is the point of running the latency tests if you're not going to use it in your conclusion?


    Nice observation. I was wondering the same thing. It's time you provide conclusion based upon what you intended to test and not otherwise. You could state the FPS part after the fact.
  • 0 Hide
    Anik8 , February 12, 2013 5:00 AM
    I like this review.Its been a while now and at last we get to see some nicely rounded up benchmarks from Tom's.I wish the GPU or Game-specific benchmarks will be conducted in a similar fashion instead of stressing too much on bandwidth,AA or using settings that favor a particular company only.
  • 29 Hide
    cleeve , February 12, 2013 5:13 AM
    ingtar33so... the amd chips test as good as the intel chips (sometimes better) in your latency test, yet your conclusion is yet again based on average FPS?what is the point of running the latency tests if you're not going to use it in your conclusion?


    We absolutely did take latency into account in our conclusion.
    I think the problem is that you totally misunderstand the point of measuring latency, and the impact of the results. Please read page 2, and the commentary next to the charts.

    To summarize, latency is only relevant if it's significant enough to notice. If it's not significant (and really, it wasn't in any of the tests we took except maybe in some dual-core examples), then, obviously, the frame rate is the relevant measurement.

    *IF* the latency *WAS* horrible, say, with a high-FPS CPU, then in that case latency would be taken into account in the recommendations. But the latencies were very small, and so they don't really factor in much. Any CPUs that could handle at least four threads did great, the latencies are so imperceptible that they don't matter.
  • 22 Hide
    cleeve , February 12, 2013 5:13 AM
    esreverWow. Frame latencies are completely different than the results on the tech report. Weird.


    Not really. We just report them a little differently in an attempt to distill the result. Read page 2.
  • 24 Hide
    cleeve , February 12, 2013 5:15 AM
    Anik8.I wish the GPU or Game-specific benchmarks will be conducted in a similar fashion instead of stressing too much on bandwidth,AA or using settings that favor a particular company only.


    I'm not sure what you're referring to. When we test games, we use a number of different settings and resolutions.
  • 12 Hide
    znakist , February 12, 2013 5:20 AM
    Well it is good to see AMD return to the game. I am an intel fan but with the recent update on the FX line up i have more options. Good work AMD
  • -6 Hide
    wh3resmycar , February 12, 2013 5:23 AM
    hey don.. where's dota 2? :( 
  • 6 Hide
    Nintendo Maniac 64 , February 12, 2013 5:30 AM
    Hey Tom's, I think I may have found a bug with the new layout. Even though this article is stated to be "IN REVIEWS", it in fact doesn't appear on the "all reviews" page:
    http://www.tomshardware.com/articles/?articleType=review
  • 8 Hide
    cleeve , February 12, 2013 5:32 AM
    Quote:
    hey don.. where's dota 2? :( 


    It's coming. Lots of other stuff to do, but it's coming. :) 
  • 18 Hide
    Nintendo Maniac 64 , February 12, 2013 5:41 AM
    Who would have thought? AMD's "MOAR CORES!" is actually paying off. :p 
  • 6 Hide
    LORD_ORION , February 12, 2013 5:44 AM
    BigMack70You know what I think this article shows more than anything? How freaking awesome the Phenom II x4 / x6 chips were for low-midrange builds for their time.


    Except they weren't cheap for their time.
    eg: 945 was $280 for its 1st year? :\

    Spend that now and what do you get?
  • -6 Hide
    wh3resmycar , February 12, 2013 5:49 AM
    can't move my wallet without dota 2 numbers. just can't.

    but then again what's making AMD hard to swallow is the abysmal TDP ratings of their APUs. hopefully you guys can explicitly explain how an a8-5500 manages 65w while a a8-5600k pulls 100w with just a 300mhz difference?

    or with power constraints, what would be more effective? an ivy bridge celeron + 6670 or a6/a8 APU? apart from the usual load/idle, what about posting-in-a-forum-power-consumption?

    i would love to post my questions on the forums but i'm pretty sure the thread'll be just ravaged by fanboys or wouldn't get a pertinent answer.


    thanks!!!
  • 3 Hide
    rdlazar , February 12, 2013 6:01 AM
    Where would the Athlon II x4 750K be on that list?
  • 3 Hide
    The-Darkening , February 12, 2013 6:04 AM
    Isn't Intel G860 13-13-13 latencies a bit too relaxed? I'm sure 10-10-10 would be possible. Any explanation for this? Just curious.
  • 5 Hide
    cangelini , February 12, 2013 6:14 AM
    Nintendo Maniac 64Hey Tom's, I think I may have found a bug with the new layout. Even though this article is stated to be "IN REVIEWS", it in fact doesn't appear on the "all reviews" page:http://www.tomshardware.com/articl [...] ype=review

    Because there are so many products, this was defined as a round-up. Unfortunately, it doesn't look like it's currently possible to use both Reviews and Round-ups as filters at the same time. So, they don't show up together. I'm going to pass this feedback back to France to see if Round-ups can be folded into Reviews.
Display more comments