HTC Opens Up The Vive Pro's Mixed Reality Potential With Early Access SDKs

HTC last week quietly released a set of SDKs that enable developers to experiment with the advanced features of the Vive Pro’s front-facing stereo cameras and audio system. The demos the company released indicate that the Vive Pro is worthier of the title Mixed Reality than Microsoft’s Windows MR partner headsets.

The Cameras

Until now, we knew little about the new camera features offered by the Vive Pro. When HTC revealed the headset, it didn’t say much about the new camera setup; we learned most of what we know about it from observation. The new cameras aren’t just for the chaperone system, they enable AR-like features, which makes the Vive Pro more of a mixed reality device than the current crop of Windows MR headsets.

The Vive Pro features two binocular cameras that face directly forward. When HTC revealed the new headset, we surmised that the cameras would enable you to see depth in the chaperone mode. We expected to learn that the Vive Pro borrows tech from the Vive Focus’ inside-out tracking technology.

In late January, Engadget reported that it spoke with Raymond Pao, HTC’s Vice President of Product and Strategy, and learned that indeed the Vive Pro would offer depth awareness in Chaperone mode. But that’s just scratching the surface of the cameras’ potential. Pao said that HTC would release a developer kit to open the camera’s functions for game interactions. Last Wednesday, HTC released an early-access version of the Vive SRWorks developer kit, which exposes that Chaperone cameras for developers to work with.

HTC revealed that the dual cameras could provide spatial mapping of static and dynamic environments. The company also released a short video that demonstrates the spatial scan process. HTC combines the data from the stereo cameras to create a 3D polygonal mesh of the environment around you. The clip shows that the process is somewhat slow, but it otherwise resembles the room-scanning process you must go through to set up the Microsoft HoloLens headset. The cameras detect walls, the floor, and objects in the room and create 3D representations of every object.

The original Vive featured a mono greyscale camera, whereas the Vive Pro features two RGB cameras that support full-color pass-through. With the pass-through view enabled and the environment mapped, you can use the Vive Pro to play with mixed reality content. In addition to tracking the environment, the Vive Pro’s stereo cameras can also track your hands like a Leap Motion camera. HTC’s hand interaction technology isn’t as sophisticated as Leap Motion’s offering, but it can manage basic functions such as reaching out and touching a virtual object.

The 3D-mapped environment enables developers to use the real world as part of your experience. You can use visual effects to alter the way the world looks, for example, such as changing the colors or textures on the wall or using visual filters that change the temperature of the ambient light. Developers can also build mixed reality content that blends with the real world. The 3D mesh of the room enables virtual objects to interact with real-world objects. The scanned room also enables occlusion of virtual assets behind real-world objects.

HTC said that the Vive SRWorks SDK is broken up into three modules: the Depth module, which handles the hand interactions; the See-through module, which lets developers work on AR/MR content; and the 3D Reconstruction module, which exposes the spatial tracking data. The SDK also includes plugins for Unity and Unreal Engine.

Audio Too

Along with the camera SDK, HTC released modules for the new headset’s audio system. The Vive Pro is supposed to have high-end headphones with a built-in amplifier, but our experience with the audio system is sub-par thus far. HTC tried to fix the Vive Pro’s headphones with a firmware update that helped, but didn’t solve the problem. Perhaps the issue isn’t the hardware, but rather the software that we’re using.

HTC released the Vive 3DSP SDK, which is a “Unity-compatible audio spatialization plugin” for the Vive Pro. The list of features in the 3DSP SDK include:

Higher Order AmbisonicsHRTFs based on refined real-world modeling (horizontally and vertically)Support for Hi-Res audio source files and playback.Acoustic distance effect with real-world modeling.The VIVE 3DSP plugin supports room effect, room reverberation and reflection, and acoustic occlusion that is tuned for the VIVE Pro. However, the VIVE and other HMDs and headphones are also supported.

The Vive Pro’s audio currently feels distant, and it’s often described as “tinny.” Maybe the acoustic distance effect and higher order ambisonics could help enhance the audio to eliminate the distant sound that the headset currently provides.

HTC also released the Vive Pro Audio Mode SDK, which includes APIs for the Vive Pro’s internal microphones. The new software enables applications to toggle the microphones between foreground monitoring, background monitoring, or one microphone for each task.

The Vive SKRWorks SDK, Vive 3DSP SDK, and Vive Pro Audio Mode SDK are available from the Vive developer resources portal. All three are early access releases.