Joshua Bell, a Grammy award-winning violinist and long-time Sony performer, partnered with Sony to create a musical experience for the PlayStation VR. The talented musician performed Brahm’s Hungarian Dance No. 1 in front of an array of 360-degree cameras and advanced audio capture equipment to create an immersive video with spatial audio and positional tracking.
Virtual reality technology promises a range of new entertainment possibilities. Video games are an obvious match for VR headsets: Games are built in digital 3D environments, which makes it relatively easy to create an experience that tricks you into believing you're there. 360-degree videos are also regularly cited as VR experiences, but that’s not exactly accurate. 360-degree video lets you look around a scene, but your head rests on a pivot. You can’t lean in to get a better look at an object or detail, for example, with a traditional 360-degree video.
Sony wasn’t satisfied with those limitations. Even stereoscopic 360-degree video wouldn’t suffice, so the company leveraged techniques used by Hollywood special effects studios to recreate the entire scene in a 3D engine to enable positional tracking. “Adding movement to this experience sounds simple enough, but actually it’s an enormous challenge,” Ian Bickerstaff, Technical Director at Sony Interactive Entertainment, said in a video about the experience.
The company used camera rigs with dual Sony FDR-X1000V action cameras to capture stereoscopic 4K video of a performance by Bell and Sam Haywood in Lyndhurst Hall at Air Studios in London. Bickerstaff explained that the team had to isolate each item in the scene, deconstruct the digital video, and then build 3D recreations of each component to recreate the video sequence in 3D.
Bickerstaff's team went to similar lengths to recreate the sound from the performance. It’s no secret that audio plays a major role in creating a believable, immersive experience. Many experts argue that the auditory experience is more important than the visual experience. If what you’re hearing doesn’t match what you see on some level, the illusion can break down. For example, if the video incorporates positional tracking, but the audio doesn’t adapt to where you’re standing, the experience won’t feel “right.”
Sony recorded the experience with several microphones surrounding the performers. The team measured the distance and placement of each microphone and used that data to map the soundscape of the experience. The performance was also recorded with an ambisonic microphone, which captures audio from all directions. This allows the sound to change as if you were really there when you lean towards one of the performers, for example.
Sony said that creating the Joshua Bell VR experience was “an enormous technical challenge.” Bickerstaff said that his team is now working towards streamlining the process and making it easier to accomplish. Sony believes its positional tracking video technology could be used for live music, theatrical performances, and sports events in the future.
The Joshua Bell VR Experience is a free experience and it's available now in the PlayStation Store.
