Basemark's VRScore Benchmark Suite: 6 VR HMDs Tested

How We Tested

Typically, we'd use our GPU test system for VR benchmarking. It consists of an Intel Core i7-5930K, 16GB of Crucial Ballistix Sport DDR4 memory, and a pair of Crucial M400 SSDs. But I wasn’t able to launch the VRTrek Test on that machine. The MSI X99S XPower AC motherboard we use does feature Realtek audio, but VRScore couldn't read a signal from the VRTrek latency testing tool. After a day of troubleshooting, I switched to an older PC with an Intel Core i5-4570K, 16GB of Crucial Balistix Sport DDR3, an Asus Z87-WS motherboard, and a 500GB Samsung 850 EVO SSD, all driven by a Corsair RM850 power supply.

Test System Specs

Image
Samsung 850 EVO (500GB)
Image
Corsair RM850

We ran VRScore with a Gigabyte GTX 1080 G1 Gaming graphics card and the PowerColor RX 480 Red Devil 8GB. The scope of this piece is to demonstrate how VRScore works, not to compare GPUs. That story is coming soon. And yes, it's going to be hot.

Test GPUs

Instead of going wide on graphics cards, I chose to include as many HMDs as possible. The line-up of platforms available to me includes an Oculus Rift CV1, a Rift DK2, an OSVR HDK 1.4, and three different Vives (our Vive Pre, the retail Vive we received for launch coverage in April 2016, and a new Vive I purchased in December 2016, which includes the new single-cable data tether).

Swipe to scroll horizontally
Test HMDsHMD ResolutionHMD Refresh RateVRScore Test Resolution
Oculus Rift DK21920x1080 (960x1080 per eye)75Hz2664x1586 (1332x1586 per eye)
Oculus Rift CV12160x1200 (1080x1200 per eye)90Hz2364x1464 (1182x1464 per eye)
HTC Vive Pre2160x1200 (1080x1200 per eye)90Hz3024x1680 (1512x1680 per eye)
HTC Vive (launch day)2160x1200 (1080x1200 per eye)90Hz3024x1680 (1512x1680 per eye)
HTC Vive (purchased Dec 2016)2160x1200 (1080x1200 per eye)90Hz3024x1680 (1512x1680 per eye)
OSVR HDK 1.4 1920x1080 (960x1080 per eye)60Hz2880x1620 (1440x1620 per eye)

We ran the System Test on each HMD, with each GPU, with and without an HMD hooked up. Basemark’s evaluation guide recommends running the VRTrek test at least five times and averaging the results. Its guidelines also suggest that you should discard any result with more than a 2% variance from the average.

Preparation

VRScore works with each of the primary VR runtime environments, enabling compatibility with Oculus’ Rift, Valve/HTC’s Vive, and OSVR’s Hacker Development kits. Basemark warns that you shouldn’t have more than one VR runtime installed at once to avoid interference. For example, if you have an Oculus Rift with the Home software and SteamVR drivers installed, VRScore may launch the incorrect runtime.

The need to remove drivers corresponding to other HMDs is an issue I wish Basemark had addressed more thoroughly. After all, uninstalling the Oculus Home software is downright inconvenient since downloaded content is tied to the Oculus install folder. Deleting that eliminates however many tens of gigabytes of content you had, too.

Uninstalling the Oculus software caused other problems, too. For some reason, after removing Oculus Home from the test system, we were unable to reinstall it. The installation routine kept kicking back an error and prompting for a reboot. This issue isn’t directly related to Basemark’s utility, but it’s a documented bug without a simple solution. We eventually had to reformat and start anew.

After reinstalling Windows 10, I left Oculus Home installed while testing one of the Vive HMDs and successfully ran the benchmark. Just remember to remove SteamVR before launching VRScore on the Rift. OSVR's runtime wasn’t affected by Oculus Home either. But again, I had to remove SteamVR to perform the test correctly.


MORE: Best Virtual Reality Headsets


MORE: Razer OSVR Hacker Developer Kit 1.4 Review


MORE: The Oculus Touch Motion Controller Review

 Kevin Carbotte is a contributing writer for Tom's Hardware who primarily covers VR and AR hardware. He has been writing for us for more than four years. 

  • Piki__
    Comparing GTX 1080 vs RX 480? HOw unfair can you be. Compare to GTX 1060!
    Reply
  • kcarbotte
    19234404 said:
    Comparing GTX 1080 vs RX 480? HOw unfair can you be. Compare to GTX 1060!

    This isn't a GPU comparison. The article's scope is to show what VRScore is all about.
    Reply
  • Piki__
    19234670 said:
    19234404 said:
    Comparing GTX 1080 vs RX 480? HOw unfair can you be. Compare to GTX 1060!

    This isn't a GPU comparison. The article's scope is to show what VRScore is all about.

    Reply
  • kcarbotte
    19234733 said:
    19234670 said:
    19234404 said:
    Comparing GTX 1080 vs RX 480? HOw unfair can you be. Compare to GTX 1060!

    This isn't a GPU comparison. The article's scope is to show what VRScore is all about.

    That is irrelevant, it would be much better seerved by including a GTX 1060 as well.

    I would have liked to, but there are two problems with that.
    I didn't have any more time to run more benchmarks. You have to run each pass 5 times per HMD. More GPUs would mean less HMDs. I cover VR, so the HMDs were my priority.

    The other problem, and this one's the kicker, is that I don't have a GTX 1060. In fact, the two GPUs that I used are the only two current generation cards that I have access to.
    Our GPU reviewers receive the graphics card samples, most of which go to Igor in Germany.
    Reply
  • thor220
    If you are trying to appear objective you are doing a poor job of it

    "Gigabyte's GTX 1080 G1 Gaming has no trouble maintaining roughly 90 FPS with the Rift and Vive, but PowerColor's RX 480 struggled to keep up."

    It should be thoroughly noted in this article that the RX 480 is not AMD's top end offering and that the GTX 1080 and RX 480 are in a different price class. Yet we have here you giving AMD the business for something that should have been obvious from the onset. While you do note that the RX 480 and GTX 1080 are in a different class you only do so in regular font in a text swamp. Something that important should at least be in bold.

    I could also nitpick how the Nvidia card is always above the AMD one or that it's charts come before the AMD ones in the 2nd half, which effects presence of mind. Likely you are projecting your own preference in this instance, perhaps without even knowing.
    Reply
  • rhysiam
    Thanks for the article, it will be interesting to see how these new benchmarks come together.

    To those annoyed by the RX 480 vs 1080 comparison, I think you're completely missing the point. The article clearly states in the title that's it's testing HMDs and trialling a new benchmarking suite. On the methodology page there's a short and very clear paragraph stating that a future article comparing GPUs is in the works (looking forward to that, btw). And it's very clearly and explicitly stated that the goal of this article is NOT testing GPUs. Then, in exploring the results of the benchmark (necessary to see how the new benchmark suite works) we see a GTX 1080 performing better than an RX 480. So what!? 480s start at less than 1/3rd the price. Maybe Nvidia fans should get angry because the 480 achieves ~70% of the 1080s performance at 30% of the price? Or perhaps you could blame the author because they're perpetuating the perception that Nvidia cards are overpriced by only demonstrating the high end (and arguably overpriced) model?

    No, this is an interesting introduction to a new benchmark, clearly labelled as such, and IMHO a worthwhile read. To be blunt, if someone just looks at a few of the charts without reading the article and concludes that Nvidia > AMD, then that's entirely their own fault.
    Reply
  • WFang
    For a review that purports to be about the HMD's, I found the lack of for example photon delay data for each HMD glaring.

    Also, this test tool overall seems to miss the mark for what I was hoping for. This does not appear to be very helpful when it comes to comparing HMD's which in my opinion is what we need more. Figuring out what hardware works best is important, but can be approximated by looking at e.g. 4k game tests already.

    As for the differences in render vs display resolution, it would be nice if it was possible to force rendering to the display native resolution and/or the lowest resolution HMD in the test round-up.
    Reply
  • bit_user
    I'm surprised they didn't just use USB 3, for the light sensor module.

    19238012 said:
    As for the differences in render vs display resolution, it would be nice if it was possible to force rendering to the display native resolution
    Until they address this, it seems the tool is useless at characterizing what a user of that HMD would actually experience.

    Apparently, 6 months' delay wasn't enough. It's still not ready for use.

    That said, I look forward to downloading it for the eye candy.
    Reply
  • sleven
    Running HDK 1.3 at 2880x1620 pixels seems to be a major handicap in this test. An overfill of 1.5x from the native 1080p resolution seems like an arbitrary setting, but I guess it's used because it was on "by default" depending on how the server was installed. Setting "renderOverfillFactor" to 1.0 would run at the HDK 1.3's native resolution of 1920x1080. 1.5x overfill is overkill for HDK.
    Reply
  • sleven
    HDK 1.4* is interchangeable with HDK 1.3 in this case
    Reply