Many people can't bear the thought of not seeing their smartphones. They'll leave them on the counter while they wash dishes, use them even after the lights dim in a movie theater, and hold them in front of their faces while they drive. So it makes sense for Microsoft to look to smartphones as the mixed reality input devices of the future in a new patent application.
Microsoft applied for the patent on February 28--after it revealed the HoloLens 2 at MWC Barcelona 2019--and it has since been published by the U.S. Patent and Trademark Organization (USPTO). It's simply called "Virtual Reality Input," and in it, Microsoft describes a system that would allow people to use their phones even when they're in virtual/augmented/mixed reality.
The company explained that current interaction methods, from hand-held controllers to gesture-based input, have their limitations. They're also unfamiliar to many people, making it even harder for them to feel comfortable in mixed reality. The solution? Allow people to use devices they use all the time--smartphones--to interact with mixed reality experiences. Microsoft explained:
"In some examples, the touch-sensitive input device may enable the reception of touch input, which may be an intuitive and familiar input paradigm to a user (e.g., as a result of repeated, daily use of a smart-phone). Touch and/or other types of input may extend the range of inputs that can be supplied beyond translation and rotation enabled by the hand-held controllers described above. The input device itself may also provide a familiar paradigm of interaction, as in some examples the input device may be, as referenced above, a smart/phone or other mobile device of the user."
While some people can navigate their smartphone without looking at it--constant use has ingrained the location of each app icon in their memory--others cannot. (It can be hard to figure out exactly what's happening on-screen with nothing but tactile or auditory feedback.) Microsoft thought of those limitations, too, and it also wants to go beyond addressing them. It explained:
"Moreover, the virtual reality experience may complement and/or extend input device functionality. As described in further detail below, the virtual reality experience may render a representation of the input device that visually indicates the device and also provides a user interface for interacting with the device and virtual reality experience. In some examples, the user interface may provide additional information and points of interaction beyond a user interface rendered by the input device itself."
It makes sense. Immersion in VR can be elusive if people aren't comfortable with a control scheme. Most people are comfortable with their phones. And if augmented reality is about building upon things in our environment, well, our phones are pretty much tied with our bodies as the most constant things in our environments. Microsoft's system merely embraces those truths in mixed reality.
