Microsoft Research Creates 'Handpose' Hand Tracking Software With Help From Machine Learning

In VR, hand tracking is one of the most important next steps in development. Leap Motion made news last month when it joined the ranks of other VR developers to continue development of its hand tracking software for OSVR. Although Leap Motion might be one of the more notable companies in hand tracking, there are others that are working on their own hand tracking software. One of these companies is Microsoft, specifically its Research wing in Cambridge, U.K. The technology is called Handpose, and even at the research stage, it already looks promising.

The demo showed one of the researchers utilizing Handpose with the help of the Kinect. However, the Kinect the researchers used has been slightly modified to better track hand movements, from a large wave to the most minute twitch of a finger.

Using the Kinect also allows the user to move around, with the only restriction being that they must be inside its range. Nevertheless, being able to move and interact in a virtual space is a highly sought-after feature in VR. With the Oculus Rift, for example, you are limited to a small surface area in which to move. However, companies like Sulon and Valve and HTC provide a more adequate amount of room for movement.

YouTube

Watch On

Getting Handpose to this point in the development process was not easy, because not only is the hand a smaller object to track than the entire human body, but it's also flexible. We can move our hands in so many different directions and angles that there are times when a finger or two is hidden from the camera. This is not to mention the fact that the camera had difficulties detecting each individual finger.

To solve this problem, the research team used the method of machine learning. By "teaching" the computer by showing it multiple images of a hand, they thought it would eventually learn how a hand moves in a space. However, this method didn't produce the expected results, so the team gave the machine an even better understanding of how a hand moves, through 3D hand models. By combining these images and 3D models, the camera was finally able to effectively track hand movements in its environment.

The developers believe that Handpose could be used in a variety of applications such as directing a fleet of robots to move in a certain direction, using the camera to interact with a screen (effectively removing the need for a keyboard), and even advancing the idea of artificial intelligence by teaching machines various hand movements and gestures.

There's also the possibility that Handpose could be one of the many technologies used in Microsoft's new HoloLens software, but at this point we're merely speculating about that. At least for now, Handpose is just an experiment.

Follow Rexly Peñaflorida II @Heirdeux. Follow us @tomshardware, on Facebook and on Google+.