Challenging FPS: Testing SLI And CrossFire Using Video Capture
What if the performance data you used for deciding which graphics card to buy was flawed? We're taking a deeper look at some of the problems with benchmarking multi-GPU configs using conventional tools. Nvidia's new FCAT suite helps us collect more info.
Multi-Card Graphics Problems, And A Solution: Nvidia's FCAT
Even now, the industry measures rendered frames per second using software-based tools like Fraps and in-game counters that capture frames from the graphics card's memory every second. On the surface, that sounds like a great way to generate accurate data.
The problem is that there can be a significant disconnect between the game engine's output and what you see on your monitor. And, when you move beyond the issues that might affect a single-GPU setup and start considering the complexity of multiple graphics processors working cooperatively, two additional variables surface, affecting your experience: dropped frames and what Nvidia is introducing to us as runt frames.
Briefly, even after a frame is loaded into memory, there's still a lot of work that goes on in the graphics pipeline; this takes time. As a result, some frames are dropped before they ever show up on-screen. Other frames show up, but only on a very small part of the screen. Nvidia's definition of a runt is any frame that shows up on fewer than 21 scan lines on a monitor.
Dropped and runt frames have no positive impact on what you see while you're playing a game. They still get counted by benchmark tools, though. This is a problem, particularly if a combination of hardware is rendering as quickly as possible without any consideration of consistency. The more drops and runts generated, the less accurate utilities like Fraps become. The good news is that these two issues only affect multi-GPU configurations enough to be a problem. In single-card testing, Fraps and the in-game metrics we use are much more accurate.
So, how do we capture frame rate data from AMD CrossFire- and Nvidia SLI-based setups without runt and drop frames interfering with the result?
We record the display output and analyze it, frame by frame. Sound intense? It is. In fact, it's so potentially time consuming that Nvidia developed its own suite of tools to make this possible.
Some of what we're using is commercially available and open source. For example, the system being benchmarked is connected to a Gefen dual-link DVI distribution amplifier, which in turn outputs to a monitor and a Datapath Limited VisionDVI-DL capture card, plugged into a second machine. For an accurate analysis, the recorded video has to be flawless, without dropped frames. So, we have a trio of striped SSDs able to cope with the uncompressed stream (as much as 650 MB/s at the card's highest supported resolution).
The system getting tested runs an overlay, which applies a colored bar to each frame coming from the graphics card in a repeating sequence of 16 colors. This software is Nvidia's, but it's necessary in order to automate the analysis. Company representatives say that the folks at Beepa could conceivably add this overlay functionality to Fraps, taking Nvidia out of the equation.
Game play, with the overlay bars, is captured by VirtualDub onto the solid-state storage. From there, another Nvidia app reads in the video and creates an information-rich Excel spreadsheet by analyzing the video's frames and looking for the color bar sequence. Knowing what it expects, the utility can easily see if a frame is dropped (color missing) or if a frame occupies fewer than 21 scan lines on the screen (a runt).
Even the Excel file is very data-heavy, though. So, Nvidia developed a series of Perl scripts able to parse the information and create frame rate/frame time analysis, along with charts reflecting the information.
Nvidia calls this complete package FCAT, for Frame Capture Analysis Tool. FCAT's resulting performance data, modified to remove dropped and runt frames from the reporting going on today, should be more in-line with what gamers see.
Current page: Multi-Card Graphics Problems, And A Solution: Nvidia's FCAT
Prev Page Frames Per Second: Why The World Was Wrong Next Page Test System And BenchmarksStay On the Cutting Edge: Get the Tom's Hardware Newsletter
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
-
kajunchicken Hopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...Reply -
cleeve kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...Reply
FCAT isn't for end users, it's for review sites. The tech is supplied by hardware manufacturers, Nvidia just makes the scripts. They gave them to us for testing.
-
cangelini kajunchickenHopefully someone besides Nvidia develops this technology. If no one does, Nvidia can charge whatever they want...And actually, it'd be nice to see someone like Beepa incorporate the overlay functionality, taking Nvidia out of the equation.Reply -
cravin I wish there were an easy way to make my frame rates not dip and spike so much. A lot of times it can go up to like 120fps but then dipts down to 60 -70. It makes it look super choppy and ugly. I know I could limit it at 60 frames a second, but wouldn't that just be like vsync? Would that have input lag.Reply -
DarkMantle Good review, but honestly I wouldnt use a tool touched by Nvidia to test AMD hardware, Nvidia has a track record of crippling the competition's hardware every chance they have. Also, i was checking prices in Newegg and to be honest the HD7870 is much cheaper than the GTX660Ti, why didn't you use 7870LE (Tahiti core) for this test? The price is much more closer.Reply
The problem i have with the hardware you picked for this reviews is that even though, RAW FPS are not the main idea behind the review, you are giving a Tool for every troll on the net to say AMD hardware or drivers are crap. The idea behind the review is good though. -
krneki_05 Vsync would only cut out the frames above 60FPS, so you would still have the FPS drops. but 60 or 70 FPS is more then you need (unless you are using 120Hz monitor and you have super awesome eyes to see the difference between 60 and 120FPS, that some do). No, the choppy felling you have must be something else not the frames.Reply -
BS Marketing Does it really matter? Over 60 FPS there will be screen tearing. So why is this sudden fuss? I guess nvidia marketing engine is in full flow. Only explanation is that Nvidia is really scared now, trying everything in there power to deceive people.Reply -
rojodogg I enjoyed the artical and it was very informative, I look forward to more testing of other GPU-s in the future.Reply -
bystander Great article.Reply
But as great as the review is, I feel one thing that review sites have dropped the ball on is the lack of v-sync comparisons. A lot of people play with v-sync, and while a 60hz monitor is going to limit what you can test, you could get a 120hz or 144hz monitor and see how they behave with v-sync on.
And the toughest thing of all, is how can microstutter be more accurately quantified. Not counting the runt frames gives a more accurate representation of FPS, but does not quantify microstutter that may be happening as a result.
It seems the more info we get, the more questions I have. -
rene13cross @DarkMantle, exactly my thinking. I don't want to sound like a paranoid goof who thinks everything is a conspiracy but a test suite created by Nvidia to test AMD hardware doesn't sound like a very trustworthy test. I'm not saying that the results here are all false but Nvidia has had a slight history in the past with attempting to present the competition in an unfair light.Reply