Skip to main content

FCAT VR: GPU And CPU Performance in Virtual Reality

Hardware And Software: Two Ways To Test

Hardware FCAT: Measuring Performance Through Video Capture

Let’s start with a walk-through of what goes into testing with FCAT through video capture, otherwise known as the hardware method.

Just like the FCAT we introduced three years ago, the new version employs a gaming system to which the HMD is attached and a capture machine responsible for recording. In between, an HDMI splitter takes input from the gaming system’s GPU and outputs two identical signals: one to the HMD and another to the capture system.

Software running on the gaming system applies two overlays to the HMD. The overlays display colored bars, one per frame, in a known sequence. Deviations from that sequence in a recorded file tell you something went wrong and a frame was not displayed. The square in the upper left-hand corner tracks warp misses, while the bar in the middle-left helps monitor for app misses. Runt frames—an artifact we checked for using the original FCAT—are no longer an issue since the Rift and Vive employ 90 Hz refresh cycles. Consequently, only full frames are displayed, and you’ll never see a partial bar.

Assuring FCAT’s accuracy requires a flawless capture at 2160x1200 and 90 Hz, without dropped or added frames. This is no small feat. While the gaming system’s specs are not integral to the benchmark’s integrity, the capture machine should be configured somewhat specifically. We’re using a Core i7-5960X on an MSI X99A Gaming Pro motherboard with 16GB of G.Skill DDR4-3000 installed. Graphics is unimportant, so we added a GeForce GTX 750 Ti to keep heat/power to a minimum.

The more important components are a Datapath VisionSC-HD4+ capture card and an Intel SSD 750 1.2 TB. Datapath’s eight-lane PCIe card supports four independent channels of HDMI 1.4 capture, two of which run at up to 297 MHz. As luck would have it, this is close to the same rate needed for the Rift’s 2160x1200 at 90 Hz. We only need one of those channels for today’s project, but that’s still a ton of data moving over the PCIe bus and onto storage. Fortunately, Intel’s SSD 750 is up to the task, facilitating sequential writes at up to 1.2 GB/s. The sample Intel sent over slides into a 2.5” form factor and attaches to the motherboard’s U.2 connector, enabling a four-lane PCIe 3.0 link.

Once the capture card’s resolution and timing settings are configured to match the HMD, and VirtualDub is set up to record at 2160x1200 @ 90 FPS, all that’s left is capturing video. The overlay is started on the gaming machine, followed by a VR application. We get to where we want to benchmark and then start recording on the capture machine (VirtualDub’s default hotkey is F5). At the end of our run, recording is stopped, leaving a massive AVI file to feed through Nvidia’s Extractor tool and subsequently analyze through FCAT’s Perl scripts.

Software FCAT: Simplifying Testing With A Local GUI

Nvidia’s FCAT software tool does away with the capture machine completely, leaving us with our gaming machine (Core i7-6700K, MSI Z170A Gaming M7 motherboard, 16GB of G.Skill DDR4 memory, and a 500GB Crucial MX200 SSD) and Rift. FCAT VR accesses the performance information provided by Oculus’ runtime logged to Event Tracing for Windows (ETW). Per the FCAT documentation, the following are measured in milliseconds:

  1. Game Start: Timestamp when Game starts preparing frame
  2. Game Complete CPU/GPU: Timestamp when Game has prepared frame (all CPU-side work is done), and then another timestamp when frame is finished on GPU
  3. Queue Ahead: Amount of queue ahead that was allowed for the frame
  4. Runtime Sample: Timestamp for warp start. Usually fixed amount of time before v-sync
  5. Runtime Complete: Warp finished on GPU
  6. V-sync: V-sync interrupt for HMD (Nvidia-only)

There are also integer counters in Nvidia’s output file to report app and warp misses. This is important functionality, and the hardware-based FCAT is limited in its ability to catch those events.

Testing on a Vive is similar, except that FCAT ties into an API exposed by SteamVR to generate its timestamps. The list of captured events is different (longer, in fact). However, we come up with the same performance statistics.

There’s one other benefit associated with software-based FCAT: Because the software is fed granular timing information from ETW and SteamVR, we can calculate what Nvidia calls unconstrained frame rate—that is, the performance we would have seen were it not for a 90 Hz refresh rate. This is particularly powerful in that we’re able to estimate headroom based on how long the render takes.

Meet The VR Capture Utility

When we first started beta-testing FCAT VR, it was referred to as FCAT 2.0. Back then, it consisted of a simple UI that let you specify a log file destination and a benchmarking hotkey. You'd fire up the capture utility and then launch your VR application. A red bar on the right side of the HMD told you FCAT Capture was running, but idle. Hitting Scroll Lock (the only hotkey supported) turned the bar green until you pressed it again, ending the run.

Not much changes in the version of FCAT VR Capture that Nvidia plans to publish. You're still able to specify a custom output directory, and Scroll Lock remains the only functional hotkey. Nvidia does, however, add options for capture delay and capture duration, making it easier to control how tests start and stop.

The first FCAT 2.0 builds we played with used Perl scripts to read in the captured data, set up comparisons, and customize the output charts. After a number of revisions (one of which was precipitated by Oculus' ASW introduction), the charts ended up looking something like this:

But all along, Nvidia knew a lot of time and effort was spent editing script files to create these charts. In an effort to encourage broader adoption of FCAT VR, the company created a Python-based GUI into which we're able to drag log files and create charts much more easily.

This VR Analyzer conveys:

  • The delivered frame rate, or what actually shows up on-screen. If you have a perfect run, you'll see 90 FPS. If your card spends half of its time at 90 and half of its time at 45 FPS, this field averages out to 67.5 FPS.
  • The unconstrained frame rate, or what you'd see if it weren't for the 90 Hz display interval.
  • Refresh intervals, or 90 times the benchmark's duration.
  • New frames
  • Dropped frames (app misses)
  • Frames synthesized by asynchronous spacewarp
  • Warp misses
  • Average frame time

All of that data paints an interesting picture of what's happening on-screen, complementing the subjective observations we relied on previously, plus the video-based capture technique we'll compare on the next couple of pages.


MORE: Best Virtual Reality Headsets


MORE: All Virtual Reality Content

  • ingtar33
    no ryzen? just an 8320?
    Reply
  • AndrewJacksonZA
    A really nice article and a great introduction to FCAT VR. Thanks Chris!

    Obvious request that probably everyone will ask of you: please include Ryzen in your tests once you have a sane moment after the hectic launch period. :-)

    Now that that's out the way, I can get to my actual question: I would like to know if "Game A" is available on both the Rift and the Vive, would FCAT VR be able to tell you what, if anything, the performance difference is given the same computer hardware please?

    Please can you also include multi-GPU setups. They can definitely help, depending on the title, e.g.
    https://www.hardocp.com/article/2016/11/30/serious_sam_vr_mgpu_nvidia_gtx_pascal_follow_up/5https://www.hardocp.com/article/2016/10/24/serious_sam_vr_mgpu_amd_rx_480_follow_up/
    I found this statement to be very interesting: "it’s possible to illustrate that 11ms isn’t an absolute threshold for rendering new frames at 90 Hz."

    Thank you
    Andrew
    Reply
  • cangelini
    Hi guys,

    Ryzen wasn't in my lab yet when this was written, but I do have plans there ;)

    Andrew, yes, FCAT VR should allow us to test the same title on two different HMDs and compare their performance.

    Multi-GPU is a plan as well, particularly once games begin incorporating better support for it (right now, that's a bit of a problem).
    Reply
  • WhyAreYou
    Thanks, looks good!
    Reply
  • playingwithplato
    Do FCAT VR's measurement algorithms tend to favor NVDIA chipsets? Curious, wonder if AMD will release a similar testing tool to measure buffer store/retrieval and render speed <11ms? Would like to see that applied to en environment with their GPUs.
    Reply
  • ffrgtm
    Fantastic article! I would love to see results with a secondary GPU (non-SLi) dedicated to physics and compare that to the cpu swap tests in AZ Sunshine you've just shown us.
    Reply
  • ffrgtm
    19429997 said:
    Do FCAT VR's measurement algorithms tend to favor NVDIA chipsets? Curious, wonder if AMD will release a similar testing tool to measure buffer store/retrieval and render speed <11ms? Would like to see that applied to en environment with their GPUs.
    A quote from another article on FCAT:

    "While the FCAT VR tool is developed by Nvidia, the company insists it is headset and GPU agnostic, and meant only to capture data. The tool itself doesn’t contain a benchmark; according to the company, the tool logs information directly from the VR runtime."

    I'm inclined to believe Nvidia's claim of brand blindness right now... but if AMD ever manages to put forth some real competition then I don't think we could be blamed for becoming more skeptical. It's hard to forget just how far Nvidia and AMD have gone to skew results in the past.
    Reply
  • AndrewJacksonZA
    19429089 said:
    Andrew, yes, FCAT VR should allow us to test the same title on two different HMDs and compare their performance.

    Multi-GPU is a plan as well, particularly once games begin incorporating better support for it (right now, that's a bit of a problem).
    Thank you.


    One thing that I might've missed in the article: Has Nvidia open-sourced FCAT and FCAT VR so that everyone can see the code, check it for unbiasedness (is that even a word? :-) and help contribute to the program to make it even better?
    Reply
  • cangelini
    Yup, check it out: http://www.geforce.com/whats-new/guides/fcat-vr-download-and-how-to-guide
    There's a download link in there. I'd be curious to hear from any TH readers who want to mess with it as well!
    Reply
  • thinkspeak
    Any chance of doing this again but with stock and OC on the maxwell, pascal and AMD cards? The 980 ti really opens up and they generally overclock well, in some cases exceeding the 1070 which would be useful for those debating an upgrading for VR
    Reply