How We Tested The Samsung HMD Odyssey
The Samsung Odyssey is the third Windows Mixed Reality HMD that we’ve had the chance to evaluate, but it’s the first one that we’ve been able to run objective benchmarks on. When we received the Acer Windows MR headset last year, we were unable to run Nvidia’s FCAT VR, and we were unable to use Basemark’s VRScore on the new HMD. We spoke with Nvidia and Basemark at the time, and both companies confirmed that their tools didn’t work with Microsoft’s VR platform. Nvidia told us that FCAT VR would eventually support Windows MR. Basemark, however, said that it didn’t expect to support Windows MR because Microsoft didn’t seem interested in helping Basemark get the benchmark working.
We were losing hope for benchmarking this platform as a whole. Nvidia took FCAT VR off its website in December, and it hasn’t yet released a new version, and Basemark has not yet released an updated build for VRScore. Faced with a limited-time evaluation loan, we once again had to resign to subjective evaluation.
Fortunately, we received a new beta copy of FCAT VR from Nvidia just in time to test Samsung’s headset. The new build of FCAT VR doesn’t officially support Windows Mixed Reality, so we can’t test Windows Store content. However, Windows Mixed Reality headsets are compatible with Valve’s OpenVR API, which FCAT VR does support. The results may not be 100% accurate, but we should be able to get a reasonable approximation of how well games run at the Odyssey’s 2880x1600 resolution.
For this evaluation, we used our Intel X99 testbench, which features an Intel Core i7-5930K clocked at 3.5GHz, a Gigabyte GeForce GTX 980 Ti Extreme Gaming, and four sticks of 4GB of Crucial Ballistix DDR4 2400 RAM. We also ran tests with an EVGA GeForce GTX 970 SC to see how well the entry-level VR-ready GPU can power the high-resolution Odyssey HMD.
|Graphics Card||GeForce GTX 980 TiGeForce GTX 970|
|Motherboard||MSI X99S XPower AC|
|Memory||Crucial Ballistix Sport DDR4-3200 (4x4GB)|
|Hard Drive||Crucial MX200 500GB|
|Power Supply||be quiet! Dark Power Pro 10 850W|
We also pulled out our Oculus Rift and HTC Vive for comparison, so that we can get an idea of how much of a performance hit the higher-resolution displays make you take.
Our Testbench features Windows 10 Professional, and at the time of the evaluation, we were running version 1709, OS Build 16299.309. We used Nvidia's GeForce Game Ready Driver 391.01 for our game test.
SteamVR has a feature called the rendering target, which is the resolution that the GPU renders the scene before sending it to the VR HMD. The rendering target is generally higher than the actual resolution of the displays, which is supposed to increase the clarity somewhat. The rendering target setting is independent of the HMD type and isn’t dictated by the resolution of the panels in the headset.
The Oculus Rift and HTC Vive both include dual 1,080x1,200 panels, and yet SteamVR’s rendering target for the two devices is 1344x1600 and 1512x1680, respectively. Steam VR treats the Samsung Odyssey, with its higher-resolution displays, to an even higher 1433x1772 pixels.
SteamVR features a slider that enables you to increase or decrease the rendering target for each HMD, which technically enables us to push the same pixel count to each device. But instead, we opted to leave the setting at the base value of 1.0 for each headset because we believe that most people would not tweak these settings before playing. We’ll explore the performance overhead of supersampling adjustments in a future article.
We selected five games to test for this evaluation, which vary in graphics fidelity and complexity: Gorn, Rick & Morty: Virtual Rickality, Arizona Sunshine, Serious Sam, and Island 359. These five titles also represent three different game engines: Unity 3D, Serious Engine, and Unreal Engine 4.
Gorn and Rick & Morty don’t offer adjustable graphics settings, so we ran those games as is. We left the preview pane in their default windowed configurations. Arizona Sunshine, Serious Sam, and Island 359 offer adjustable graphics settings, and we normalized the parameters for each headset and GPU configuration.
For Arizona Sunshine, we set each video setting to the maximum value, including draw distance and texture quality. Serious Sam VR features pre-set graphics configurations, which are dictated by the hardware in your system. We hit the auto-detect button with the GTX 980 Ti installed, which set the CPU speed to ultra, GPU speed to medium, and the memory clock to ultra. Each time we changed a variable, we set the configuration back to the original auto-detected settings. In Island 359, we set the graphics to the high preset, which sets every parameter to high and the draw distance to 130.
For each test, we captured the framerate data during 60 seconds of gameplay, then compared the results in Nvidia’s FCAT VR Analyzer. We ran each test multiple times to rule out anomalous behavior.
MORE: Best Virtual Reality Headsets
MORE: All Virtual Reality Content
MORE: Virtual Reality Basics
Why? Is it the inferior Vive controllers? The sweet last-gen outside-in tracking? Displays? The extra $100 for the DAS? When you make a comment like that, it's nice to point out why.
I believe there was a fairly glaring omission from your otherwise thorough review.
WMR requires Bluetooth 4.0. As BT is more prevalent on laptops than desktops, that might be a gotcha when people get their WMR unit, regardless of manufacturer & model.
"Cool! It's finally arrived!".... "Wait, WTF...I need a BT adapter??? GAHHH!"
Here's a couple of tips that I didn't note in my own review from my month+ spent with the Odyssey.
1) Before anyone purchases any WMR device, you should run Microsoft's WMR PC Check tool. It's a free download in MS' App Store. (HERE) If there's any "X" that it calls out as a problem, WMR will not work. Period.
2) This is the BT adapter that MS officially(?) endorses on their website. Link goes to product on Amazon.
3) I found the controller tracking to be very inconsistent with the Odyssey. In your gaming/testing, did you do much movement where the controllers needed to go to the outer stretches of your arms or behind your head (for games such as Space Pirate Trainer and Beat The Blitz)? My controllers with fully juiced batteries would lose sync which really detracts from the total immersion.
Also, you are correct about the battery cover. As I noted in my review, my grip on the controller was dislodging the battery cover.
4) If my understanding is correct, the controllers pair to your BT adapter. This potentially causes signal attenuation if your BT adapter is plugged into a rear USB port or is simply not always in line-of-sight with the controllers.
5) I too found the tether to be too short. So I ended up buying some extension cables which resolved the problem. Be aware, not all extension cables are created equal. For example, a 'standard' USB extension cable does not infer USB 3.0 compatibility. And WMR is very specific if you aren't plugged into a a USB 3.0 port. I thought maybe it was just power delivery, but I was initially using a powered USB 2.0 extension cord. No go. It has to be a USB 3.0-rated extension cable. Same thing with HDMI certifications.
So here's what I recommend for people looking to extend their playing area. (Obviously, I confirm these work with the WMR headsets, specifically the Odyssey)
- 6' HDMI extension cable- USB 3.0 extension cable- Any run of the mill USB 2.0 extension cable so get your BT adapter closer to the play area.
6) Lastly, with regards to the comparison of built-in audio on the Odyssey compared to the lack-of on the Vive... The Vive has the 1/8" headphone tap on it. A solid set of earbuds fill the need nicely. Their cable drapes over the back of your head and are never in the way. So people don't have to resort to placing big, over-the-ear headphones over the Vive's head strap. I will say that the AKGs on the Odyssey do work exceptionally well!
Again, a nice thorough review. I just wanted to help plug what I consider to be a few holes so the readers can avoid a lot of the delays and headaches that I initially dealt with.
And I also agree- especially given the Vive's price drop since the Vive Pro released- the Vive is still the overall HMD winner.