A Taste Of Tobii's Foveated Rendering And Head Tracking

For 2017, Tobii announced that it plans to bring foveated rendering technology to VR and PC gaming. We expected the company to have a working prototype of the technology in time for shows such as GDC or E3 later in the year, but Tobii had a working demo ready for CES. In addition to the early surprise, we were also able to try out the company’s latest tracker that comes with a new Acer gaming monitor.

Rendering The Future

Simply put, foveated rendering is a technology that helps reduce the GPU load for virtual reality experiences. To do this, however, you need eye-tracking software that makes sure the GPU only renders the images in your field of view. The demo Tobii used to show it off--which wasn't in VR--provided an example of how this new technology works. On the screen were three curtains of different colors, as well as two flower pots in the foreground. I was told to look at one of the two flower pots for a few seconds and then look at the other pot to see the feature in action.

The result is nearly instantaneous, but you can still see the rendering process at work. The flower pot that held my attention was clear and fully rendered, but everything around it seemed blurred out. When I looked at the other flowerpot, there was a split second where it was still blurry, and then the GPU rendered the image. The effect is similar to when you change a camera's focus to make the image clearer, except foveated rendering will (potentially) perform this feature multiple times throughout a game or VR experience. We even saw a quantitative sample of the amount of GPU resources saved due to foveated rendering. The fully rendered image was over 3MP, but with the new rendering feature, that was reduced to just 1.73MP.

For an early prototype, Tobii’s attempt at foveated rendering is off to an impressive start. However, the demo used a scenario that didn’t have any moving objects. It would be interesting to see how the same rendering feature would handle fast-moving sequences, but for now, we’re content with what Tobii showed off.

Use Your Head

Acer revealed three new monitors in its Predator gaming lineup, and one of them featured Tobii’s IS4 platform, which provided the traditional eye-tracking technology along with other features, such as head tracking. The IS4 was initially announced in 2015 and was considered to be the first platform from the company that met specific requirements so that it could work with consumer devices. In this case, it was built beneath Acer’s new Z301CT monitor.

To get an idea of its performance, I flew around a planet in Elite: Dangerous. I used my eyes to pan across the console of my spacecraft, but if I wanted to take a broader look at my surroundings, I turned my head so that I faced the left or right edge of the screen. The camera response had minimal latency, so it almost felt as if I actually moved my head within the game.

It took a while for me to get acquainted with eye-tracking controls when I first, er, saw them, and I had a similar experience with this demo. After a few minutes, I finally felt comfortable flying my ship while using my head to control the direction of the camera.

Initially, the idea of using my head to control the camera seemed marvelous, but the more time I spent with the tracker, the less it felt like natural movement. I “cheated” the tracker by using my eyes to move the camera to a specific direction and then turned my head to move it even further. In the real world, this isn’t how you turn to look in a different direction. Your head moves first, then your eyes will process what you see. It’s also worth noting that you can’t turn your head at a rapid speed because the tracker won’t fully process the move.

By no means is the tech perfect, but at least it’s a glimpse of what’s possible with the Tobii’s technology.

The Work Never Stops

Even though both features still need more work, the fact that we were able to see them in action and without too many problems means that Tobii is doing well in improving its tracking technology in PC gaming, and more importantly, in virtual reality. Head tracking isn’t an issue in VR, but the fact that you can have a device that mimics the feature without a headset is impressive.

However, the main attraction is foveated rendering. When it’s implemented into VR platforms, which already require a significant amount of GPU resources, foveated rendering could significantly reduce overhead, which could also lead to cheaper, more affordable PC requirements for a VR headset (or internal specs in the case of self-contained all-in-one HMDs). Tobii plans to deliver a completed version of its foveated rendering technology this year.