MindMaze developed a cost-effective VR HMD upgrade called Mask that can detect biosignals in your face to bring your facial expressions and emotional queues into VR experiences to help improve virtual social interactions.
Social VR experiences, which let you communicate and share activities with other people from around the world, are incredibly compelling. Oculus proved that when it showed off Toybox for the first time, and platforms like AltspaceVR, VRChat, and Rec Room as well as co-op games like Arizona Sunshine and Raw Data reinforce the idea.
In the real world, emotional expression is a large part of face-to-face communication. You can tell a lot about a person’s demeanor and intent from the expressions on their face, irrespective of the words they are saying. In VR, you don’t have the luxury of seeing the face of the person you’re talking to, so you can’t pick up on such emotional cues. The developers of VRChat mapped several facial expressions to the thumbstick/trackpad on your controller to address this problem, but those expressions aren’t natural, which means they aren’t a reliable gauge of that person’s mood.
MindMaze has a solution called Mask that could bring emotional communication to VR, which could make social VR a whole lot more interesting.
“Humanizing virtual reality is a natural progression. What makes us human in social interactions is emotion, but that’s been missing in virtual reality so far,” says Tej Tadi, CEO, and founder of MindMaze. “For the industry to progress, all experiences, but especially those in transmedia VR, have to improve. Until now it’s been stunted by the lack of quality, immersive content. MASK brings emotion, the ultimate driver of engagement, with the goal of evolving VR beyond novelty and into the mainstream.”
MindMaze Mask is a combination of hardware sensors and software. Mask uses biosignal processing technology and machine learning from the healthcare industry to detect your facial expressions “tens of milliseconds” before they manifest on your face. MindMaze’s software then replicates your expressions on a VR avatar in real-time.
Mask can replicate a wide variety of facial expressions, including anger, disgust, happiness, or sadness. MindMaze said that Mask’s sensors detect electrical impulses in your face to determine your facial expression. The company developed proprietary algorithms to analyze the signals and build a “neural signature” of your expressions.
MindMaze engineered Mask technology to be an affordable upgrade for existing VR HMDs and to be compatible with any HMD on the market. The sensors that MindMaze uses for Mask are compact and don’t require a lot of power. The company said they would work well in mobile VR devices.
We spoke with Tej Tadi, MindMaze’s founder and CEO, via Skype shortly before the Mask announcement. During our call, Tadi showed us a remote demo of the Mask technology in action, and honestly, our jaw dropped to the floor when we saw the technology in action. Tadi had a volunteer with an upgraded OSVR headset demonstrate the facial expression tech, and to prove the demo was real, they asked us to request expressions. During the test, they showed us smiling, frowning, and a look of surprise, and the transitions from one expression to the next were seamless and instantaneous.
We also asked to see a single eyebrow raise, but Tadi said that wasn’t possible with the current number of sensors. Tadi said that a larger number of sensors would allow tracking of such an expression, but he wants to keep the costs down to a minimum. He didn’t specify how much a Mask upgrade would cost, but he said the setup costs “tens of dollars” to build and he plans to keep it that way.
Tadi said MindMaze is currently shopping for partners to bring the Mask product to market. The company is considering fostering direct relationships with HMD makers, but it's also exploring an aftermarket solution that could work with any VR HMD. Whichever path the company takes, we just hope it doesn’t take long to bring Mask-enabled hardware to the market.
This technology relies on direct contact sensors, so I doubt it could be a software-only upgrade. But I disagree with your assessment. The upgrade wouldn't be very expensive to add to any HMD, and MindMaze is looking for partnerships with HMD designers so the sensors could be part of the design of future headsets.
$10/HMD is a lot, in a rapidly commoditizing market.
The question really is whether this is a must-have feature. If not, I think it'll be hard to justify. Especially if only a few people have it.
The main thing that could sell it is if VR-based teleconferencing catches on and you need some tech like this to get decent mouth movement. I can see the benefits of having quality mouth movements and facial expressions, if you're spending lots of time talking to people in VR.
BTW, one thing I wonder about is how clean one must keep the sensors. Do they need any type of gel? That'd be a deal-breaker, for most people.
Telepresense will likely become a big part of most people's every day live's in the coming years, so I would say that, yes, facial expressions are a big deal in VR. $10 per HMD as a prototype doesn't sound like a lot to me. As with anything, prices come down over time. By the time this tech is ubiquitous the costs will be much lower.