In advance of the 2017 Game Developer Conference (GDC), SensoMotoric Instruments (SMI) revealed to Tom’s Hardware that it is collaborating with Valve to bring its eye tracking technology to Valve’s virtual reality platform. The two companies succeeded in pairing SMI’s eye tracking system with HTC Vive headsets, and now they're integrating the technology directly into Valve’s open (as in, anyone can use it) VR platform, OpenVR.
Valve and SMI installed SMI eye tracking technology into a handful of HTC Vive HMDs and shipped them to research partners around the world so they can start experimenting with the possibilities offered by tracked pupils. Last week, Google revealed its mixed reality “Headset Removal” technique, which is possible thanks to SMI’s upgraded Vive headset.
“Eye tracking opens up several interesting possibilities to both VR developers and customers,” said Yasser Malaika of Valve. “Our collaboration with SMI on R&D, as well as on SMI’s efforts to make eye tracking enabled Vive units available to the larger VR community, have been critical to our growing understanding of how HMDs with integrated eye tracking will positively impact the future of VR.”
Eye tracking isn’t a new concept. We’ve seen implementations of the technology in laptops, and you can buy devices that bring eye tracking tech to your desktop PC. Eye tracking is somewhat niche for flat displays, but it’s one of the holy grails of virtual reality advancement, and it is sought after with vigor.
In 2015, Fove Inc. launched a Kickstarter campaign to create an eye tracking focused VR developer kit (which started shipping in January). Later that year, Starbreeze revealed it was working with Tobii to bring eye tracking technology to the StarVR HMD (which you can try at IMAX in LA). SMI also has agreements in place to put its eye tracking tech in at least 10 upcoming HMDs, including mobile VR and AR devices powered by Qualcomm’s Snapdragon VR820, and it's a key component in Qualcomm's VR HMD Accelerator Program and Snapdragon 835 VR dev kit.
There’s good reasoning behind the push for VR with eye tracking. Today’s VR experiences are incredibly compelling, but eye tracking technology stands to increase the sense of presence by a large margin. By tracking your pupils, digital artists can create virtual characters or virtual avatars that make eye contact with you in an experience like VRChat. Eye tracking also lets you interact with items and menus at a glance, instead of with a click of your mouse, flick of your wrist, or press of a button. The technology is already used in VR HMDs for the medical industry to detect brain damage and augment therapy for paralyzed patients.
Foveated rendering is the big-ticket item on the list of eye tracking enabled technologies because it could be the gateway to higher resolution VR HMDs. In fact, according to Michael Abrash, Chief Scientist at Oculus, we may never reach 4K resolution per eye and beyond without foveated rendering.
The idea behind foveated rendering is simple: It reduces the GPU workload by lowering the quality of the image outside of your focal point. When you move your eyes around, the eye tracking hardware relays your pupil position to the render pipeline, and the image adapts accordingly. The section that you are focusing on gets rendered at the highest fidelity, and the area in your peripheral vision runs at reduced quality settings, which results in higher framerate performance.
Technology that can lower the graphics demands of VR content is critical for advancing the average resolution of VR HMDs, but it can also help improve the experience you get with existing VR hardware. For example, developers could use the freed-up resources afforded by foveated rendering to push the graphic details of their game higher.
Valve will have SMI’s upgraded HTC Vive HMD at GDC 2017 to give demos of the new OpenVR eye tracking features to developers and members of the press.
“[…] We are thrilled to see our eye tracking on show as part of the Valve platform,” said Christian Villwock, SMI Director OEM Business. “This demo is the result of the experience and the valuable learnings we have accumulated during our relationship with Valve, a company that had the foresight to see the value of eye tracking at an early stage.”
Update, 2/28/17, 12:00pm PT: The original story incorrectly stated that OpenVR is open-source. It is not open-source, but you do not need a license agreement to use it.
