It’s still the early days of VR, and although Oculus and HTC seem entrenched already as the major VR hardware makers, there’s much left unsettled. To wit: Processing power for VR is at a premium, and some technologies are still largely missing from many VR HMDs, such as eye tracking. SensoMotoric Instruments (SMI) has a plan for that.
Foveated Rendering
SMI has developed a solution that offers foveated rendering in essentially any HMD. It uses eye tracking to accomplish this feat, and the eye tracking is performed by a pair of tiny cameras attached to a minute PCB, all of which is embedded in a VR HMD.
Simply put, foveated rendering saves processing resources by focusing primarily on the center part of what you see in a VR experience. Essentially, it renders in full only what is in your field of view, and it provides the lowest latency and highest framerates for the small area where your eye is focusing at any given time.
It makes sense, right? If you aren’t looking at a certain area of a VR experience, why render it and waste precious GPU resources? But it’s easier said than done.
In order for foveated rendering to work, the system has to know where you’re looking and also adjust accordingly. Eyeballs can move quickly, so you need an eye-tracking solution than can read those movements fast enough and tell the system to adjust, and the system, then, has to perform the adjustment. All of this has to happen in the blink of an eye, as it were.
Rapid Eye Tracking: Undefeatable
There are some companies working on eye tracking for VR HMDs, including QiVARI and Fove, but SMI’s version was impressive for a couple of reasons: For one thing, I couldn’t defeat the foveated rendering no matter how hard I tried. For another, the cameras and assembly are designed such that they could fit into almost any HMD, mobile or not, and the whole thing costs less than $10.
The cameras look straight through the HMD’s lenses, resulting in what SMI called full field of view, and it works with all types of lenses. There are two circles on the display to show the foveated rendering (for demo purposes only). The inner circle is rendered at 100%, the outer circle is at 60%, and everything beyond that is rendered at just 20%. SMI claimed the module can deliver the image at 250 Hz.
When I tried the two demos SMI had running, of course the first thing I tried to do was catch the foveated rendering in the act. One demo used an Oculus Rift DK2, and the other used a Gear VR.
Using the Rift, I went through the quick calibration protocol (you look at a red square three times, and it takes about as long to perform as it does to read this sentence), and then I entered a virtual room -- kind of European and rustic-looking, with a staircase and a black and white tile floor -- and I started darting my eyes around as quickly as possible, and I even shook my head from side to side to try and spot the unrendered (or under-rendered) area. It was to no avail; the system was too fast for me.
The Gear VR has the same quick calibration exercise. The demo SMI showed us consisted of three games, one of which was a whack-a-mole game (that I’m proud to say I conquered in significantly less time that our Editor-in-Chief, Fritz Nelson, did). You simply look at wherever you want your hammer to drop (there’s a small cursor that lets you know where you’re looking), and you press the trigger on a small handheld controller to execute. In another game, you look at boxes and shoot at them such that they fly up into the air; your task is to keep shooting and sort of juggle them.
In both, I was surprised at how fast the eye tracking worked. It was almost as if my eyes and the tracking system were moving faster than my brain could process what was happening. (Go ahead, insert joke here. It’s okay, I’ll wait. All done? Great, let’s proceed.)
In the final game in the Gear VR demo, you’re in a dark room, and wherever you look is where the "flashlight" illuminates. In this case, the "cursor" is a narrow beam of light (it’s a clever way to dress up the eye tracking cursor), and you can engage with items by looking at them and clicking a button on the controller. Here, you can interact with most of the objects in the room, including the crackling fireplace. You can actually fan the flames, and the poetry was too great for me to resist doing so.
To put it colloquially, the eye tracking works lickety-split. As I experienced in the QiVARI demo I tried at CES, I would have actually liked to dial down the resolution somewhat in order to gain a bit more control.
The Catch-22
SMI is well aware of the catch-22 of these demos: If the foveated rendering functions properly, you can’t tell it’s working. Therefore, in terms of what you actually see, the demo would offer the same experience with or without foveated rendering engaged. Although SMI didn’t have any way to show the effects of foveated rendering on the Gear VR demos, the team did have the Rift demo running a simple CPU/GPU usage and FPS meter.
With the Rift running the demo I described above, the CPU and GPU were at a certain level. When the demonstrator flipped foveated rendering on, the GPU usage immediately dropped from about 70% usage to 50%. He turned it on and off several times, and the results were about the same -- and instantaneous -- every time.
For now, we suppose we’ll just have to take their word for it on the Gear VR; SMI said that the difference in CPU/GPU usage on a mobile phone with foveated rendering is far more stark than on a PC that can push tons of processing power no matter what the experience demands. (That’s why the FPS didn’t change in the Rift demo. However, the point is that the system didn’t have to work as hard to deliver the same FPS.)
Not Another HMD
To be clear, SMI’s aim is not to make yet another HMD. This is partially why the team showed its eye tracking on existing HMDs instead of using their own prototype/proof of concept mockup. No, SMI wants to get into other companies’ HMDs. (Except for OSVR, which apparently doesn’t have enough space for the tiny cameras and module to fit.)
The value proposition is certainly there. Neither the Rift nor the Vive have eye tracking nor foveated rendering, and this add-on would increase the cost ($600 and $800, respectively) by just a few dollars. Further, although SMI doesn’t seem to be too keen on the idea, this inexpensive stereoscopic camera module would be a strong potential pairing with an inexpensive Google Cardboard HMD, as the whole package could cost as little as $30.
It's also potentially ideal for AltSpace VR, a company with which SMI has been working. In AltSpace VR, you have an avatar in a social VR space. With eye tracking, you could actually gaze into the virtual eyes of another person's avatar, creating a more powerful experience.
MORE: Virtual Reality Basics
Seth Colaner is the News Director for Tom's Hardware. Follow him on Twitter @SethColaner. Follow us on Facebook, Google+, RSS, Twitter and YouTube.
