Intel Adds Hand-Tracking To Project Tango Phone, Without Peripherals

Back at IDF 2015, I saw a couple of demos with Intel’s Project Tango smartphone, and although I could see the potential for what the group was trying to accomplish, the whole thing was a little rough.

In just a few months, the group inside Intel working on Project Tango has added at least one major update, in the way of hand and object tracking, and the Intel team did it without using any additional peripherals. At IDF, they showed hand tracking with an OSVR headset and a Leap Motion sensor mounted on the front. Now, they can do it with just the RealSense camera (ZR300) embedded into the phone, with six degrees of freedom (6DoF).

To be more specific, project head Paul Zhao calls it “hand virtualization,” not “hand tracking,” but the effect is the same. You can see your hands rendered in the virtual environment. If you get the device’s camera close enough, you can see a facsimile of any walls or other people you might bump into, as well.

HMD And No HMD

There are really two ways to use a device like Intel’s Project Tango phone. One way is as an augmented reality device, where you hold up the camera, and through the phone’s display you can see various things superimposed on the real world. Another way is to drop the phone into an HMD and enter a VR environment.

Zhao showed me a simple HMD that the Tango team whipped up, and he ran me through a couple of demos. Mainly, I was presented with a mountainous environment and my virtualized hands extended out in front of me. I could walk over to a stone structure and knock it over, put my hand under a waterfall and watch the water splash off of it, and walk off the edge of the cliff. (I didn’t, though, because I could not force myself to do it, and furthermore, when I edged close to the precipice, I bumped into a real physical wall, and after that I was done pushing my luck.)

The hand tracking virtualization was far superior to what I saw at IDF, but there’s still some work to do. As you can see from the images, the facsimile hand is quite pixelated, and you have to keep your hands extended far out in front of you so that the RealSense camera can see them. With slower movements, the tracking was persistent, but I’m not confident that it will be strong when you move more quickly. (To be honest, I didn’t even think to test that out, because when I had the HMD on, I was in an area with lots of foot traffic and nearby walls, and I didn’t want to whack anyone.)

There’s a two-pronged approach to achieving this tracking, one active and one passive. The first part consists of laser projection (active); lasers bounce off of objects to let the camera know where they are. However, this leaves “holes” in the picture. Zhao noted that although they could send out a shotgun blast of lasers to get a more complete picture, what ends up happening is that you get too much noise and false positives. Therefore, they limit the number of lasers and instead employ “ambient IR” (passive) to fill in those holes without introducing all the noise.

Image 1 of 2

Image 1 of 2

Intel Is In The Camera Business

Intel’s Project Tango team deserves some acclaim for making big strides in a relatively short time.

They’ve built a phone and added the ability to produce virtualized versions of your hands, other people nearby, and planes such as walls and floors, and they’ve done it with an embedded RealSense camera instead of a peripheral. To enable VR experiences, they made their own HMD, too.

Even so, the phone is very much a phone; Zhao told me that the device has passed FCC certification, so you could drop a SIM card into it and use it like any other handset. And the whole package, including the final version of the dock I saw at IDF and the SDK, costs $399. It’s shipping in Q2, and if you want the dev kit, you can get it here.

But really, Intel doesn’t want to sell phones. It wants to sell RealSense cameras. Its Project Tango phone is an elaborate proof of concept for what RealSense can do on a mobile device with Google’s Project Tango technology (or a variety of others, such as Unity). It hopes that its relatively inexpensive dev kits will encourage developers to innovate on the hardware.

It’s been rather slow going and somewhat up, and down, for RealSense cameras across the board, but Intel’s handset shows some of the capabilities it unlocks -- including what it would be like to have both AR and VR capabilities baked into a smartphone.

Seth Colaner is the News Director for Tom's Hardware. Follow him on Twitter @SethColaner. Follow us on Facebook, Google+, RSS, Twitter and YouTube.