Microsoft Expanding Visuals Beyond TV with IllumiRoom

On Monday during the Association for Computing Machinery’s 31st Conference on Human Factors in Computing Systems (CHI), Microsoft introduced IllumiRoom, a means of expanding visuals beyond the physical limits of a screen by taking advantage of surfaces in the surrounding area. This is accomplished by integrating a Kinect sensor with an off-the-shelf InFocus IN126ST projector to extend the field of view.

To better explain this, imagine an HDTV seated within an entertainment center with shelves on the left, shelves underneath, and several on the right with a couple of plans on top. This proof-of-concept system can recreate the entire scene and project it across the actual physical objects. Thus, if the viewer is playing an FPS on the Xbox 360, a gunshot can cause an emulated ripple effect across the entire scene (AKA distort reality).

"Our system can change the appearance of the room, induce apparent motion, extend the field of view, and enable entirely new game experiences," Microsoft states. "Our system uses the appearance and the geometry of the room (captured by Kinect) to adapt the projected visuals in real-time without any need to custom pre-process the graphics."

The prototype can connect to existing game content even if the game’s source code is not available. It also captures a game’s image by using a video-capture card, then employs optical-flow techniques to analyze images and drive the illusions. The Kinect sensor itself captures the color and geometry of a scene and develops a depth map of the space, allowing the system to be used in any room.

"From this first step, correspondences between the projector and Kinect’s color camera were transformed into 3-D points of reference that provided a careful calibration enabling a tight match between on-screen content and the physical environment," Microsoft reports. "The central challenge, though, was generating real-time, interactive illusions that would adapt automatically to any room, any display, and any game."

The team behind the prototype didn't want to just extend the FOV of a video game, but to also selectively render scene elements to augment the appearances of the physical environment (like the gunshot example). They tested eleven effects on the physical environment ranging from super-saturated colors for a cartoon look, to de-saturated color schemes that created a film noir look.

"So far, we’ve made the experience spill out into the room, but in the future, we’d like to make it much more interactive," says project leader Hrvoje Benko, a researcher with the Natural Interaction Research group. "For example, if a ball falls out of the screen, we’d love to be able to throw the ball back! Plus, we’d like to understand what new cinematic effects are possible when content is split over two connected screens."

Microsoft describes one mode as Focus+Context Full which spills the game's field of view across the entire physical space. In Focus+Context Edges, the FOV is still extended out beyond the screen's physical limits, but only as white lines on a black background. In Focus+Context Selective, only certain elements escape the TV frame like weapons fire and explosions. There's also Focus+Context Segmented to make it appear as if the game is playing on the wall behind the furniture.

"IllumiRoom can change the appearance of the room to match the mood of the on-screen content," Microsoft explains. "For example, one can saturate the room colors, make the room appear back and white, or highlight the room's edges. By distorting and re-projecting the room's texture, it is possible to warp reality and make it appear as if the room itself is responding to the game."

Along with revealing the prototype, the team has launched a 10-page paper entitled "IllumiRoom: Peripheral Projected Illusions for Interactive Experiences" which can be read in PDF form here. There's no question that this system will be a future peripheral for the Xbox Infinity, but when that will be is unknown at this point.

We've added one video demonstration below, but additional videos can be accessed here.