Tobii’s Integrated VR HMD Eye Tracking: Hands-On

Tobii has made large strides in bringing its eye-tracking technology to VR HMDs. The company gave us our first look at its HMD integration in action at GDC, where Tobii showcased a fully integrated eye-tracking sensor in an HTC Vive.

A Noble Cause

Tobii’s goal within the realm of VR is to bring the experience to new levels from the standpoint of both the developer and the gamer. Eye tracking can be leveraged to implement foveated rendering techniques (a trick Tobii showcased on a 2D monitor at CES), reducing the host PC platform’s workload. Reducing hardware requirements for VR applications is an effective way to bring the technology to an affordable mainstream level, and HMD eye-tracking integration is the first step of that venture. Although the eye-tracking technology can enable foveated rendering, the demo we saw didn’t use this feature.

Accurate eye tracking can also provide better 3D stereoscopic rendering by adjusting the image to appear correctly based on the user’s eye positions. This could prevent aliasing and artifacts caused when the HMD drifts on your face. We saw this in action with the integrated sensors in the Vive, and the image was crystal clear without hardly any adjustment to the face, even if the HMD was slightly askew.

The HMD eye-tracking integration that Tobii showed us featured the company’s proprietary EyeChip, a dedicated ASIC that reduces the workload of the host PC’s CPU by handling the sensor data on its own. It also reduces the peripheral’s power consumption, and the same EyeChip is found in the Tobii Eye Tracker 4C (the company’s latest generation product), which operates at a frequency of 90Hz. However, the sensors in this Tobii-integrated Vive are located behind the lenses, and they are able to track your eyes through them.

You Can’t Miss

Tobii’s demo software provided several use cases to improve the user experience with HMD eye-tracking integration. The most obvious benefit is natural targeting, which enables users to pick up, aim, or throw an object simply by looking at the desired target.

To see this feature in action, Tobii’s demo placed me in a field with some rocks at my feet and some bottles precariously placed on various surfaces in front of me. First, I picked up a rock and threw it without the eye tracking targeting enabled, and needless to say I couldn’t hit a thing despite my meticulous attempts at lining up my shot. Next, I grabbed a pink-illuminated rock that had the eye tracking enabled, and I couldn’t miss a single bottle so long as I looked at it.

Look Into My Eyes

Being able to track and display your eye movements on an avatar further increases the immersion of VR by adding the human element of expression. Basic eye contact is a social interaction that can convey so many different emotions and thoughts, and representing that in a VR environment is essential to creating authentic human interactions, either with other users or NPCs.

At one point, Tobii’s demo allowed me to look into a set of mirrors with an avatar, with one of them reflecting the animated character skin without the eye tracking enabled. The other mirror had eye tracking enabled, and my eye movements (including the direction of my gaze, blinking, and winking) were captured and applied to my avatar with impressive accuracy and virtually no latency.

Another location within the demo showcased NPC interactions that become possible with eye tracking. A pair of robots would divert their attention to me by turning and looking at me directly if I was looking at them, and again, the accuracy of my gaze was well-represented in the VR environment. Each robot had its own purpose (one gave me money and the other sold a variety of objects I could throw around, including rubber ducks, beach balls, and sand buckets). Again, eye tracking was critical to grabbing objects at a distance.

Develop Now, Play Later

Tobii revealed that it brought its integrated eye tracking solution to GDC to spark interest in the developer community. The company’s partnerships have gotten it this far, but more work is required to truly determine the limits of the current technology and its best practices of implementation. Although Tobii isn’t the first eye tracking sensor to make it to a mainstream HMD--SMI demoed its inexpensive eye tracking sensor and foveated rendering with an Oculus DK2 and GearVR early last year--it’s probably the first company to showcase a fully integrated sensor system in one of the two major PC-dependent retail version VR HMDs.

Tobii said it plans to feature its fully integrated eye-tracking sensors and EyeChip in the next generation of mainstream VR HMDs, implying that we would see them as early as the end of 2017 or early 2018. At present, we can say that Tobii’s eye tracking delivers on its goal of a more immersive experience, improved social interaction, and easy targeting. If developers come on board to create content for it, Tobii Eye Tracking (and eye tracking in general) could become a must-have feature on future HMDs.