I recently attended Immersed 2015, and after going through my notes, an obvious recurring theme emerged: Many, if not most, of the speakers and panelists at the event were of the same opinion that the auditory experience in VR is just as important as the visual experience -- if not more important.
"Sound is 50 Percent Of The Movie Experience"
There were many presentations at Immersed 2015, and if I learned one thing from them, it is that the people working diligently to bring the best, most compelling virtual reality experiences to fruition have all realized that audio can't be ignored, though it largely has been thus far.
Many of the early VR experiences that are available for the public to download and play have been unconcerned with the auditory experience, but a bad one can completely break the immersion. George Lucas once said, "Sound is 50 percent of the movie experience," and for virtual reality content, this is probably even more true.
One of the early presentations at the conference was "Working With Immersive Audio For Virtual Reality" by Jan Nordmann, Senior Director Business Development, New Media at Fraunhofer USA (inventors of the MP3 codec and co-developers of AAC). His company is working on new audio standards for immersive 3D audio in virtual reality and has already partnered with Samsung on the Gear VR.
Nordmann said that people perceive where sound is coming from based on the delays heard in either ear. In virtual reality, this becomes incredibly important, especially in things like virtual cinema. Imagine looking away from a TV screen but the audio still sounds like it's right in front of you.
The way that noises sound is important, as well. If you leave an office building and head outside into a busy street, it shouldn't sound like you are still inside the building. This type of thing can break immersion quickly.
Fraunhofer has developed a technique called Head Related Transfer Function (HRTF), which uses binaural sound recordings to simulate the direction sound is coming from. The process to record sound this way requires two microphones placed where a person's ears would be, with speakers around the room delivering audio from all directions. Despite being recorded with multiple speakers, Jan believes that the typical VR experience will require headphones for total immersion.
Movies Need Surround Audio, Why Wouldn't VR?
On day two, there was a discussion panel called "The Audio Of VR" that was moderated by Jason Jerald, Principal Consultant at Nextgen Interactions. Jan Nordmann was one of the two panelists. He was joined by Mary Spio, President of Next Galaxy.
To start off the discussion, Jerald brought up a great point: Most people would consider watching a film without sound to be a ludicrous idea these days, so why do we see so many VR demos without audio? He said this makes the user deaf.
The answer to that question seems to be that audio has been treated as an afterthought in the quest for pushing virtual reality to fruition. Companies developing both hardware and software for VR have been spending their time and resources on solving the visual problems first. Based on my experience at Immersed, though, it appears that industry insiders are starting to make the shift to audio immersion.
The first generation of immersive audio technology is focused on getting the head tracking working in the audio environment. To keep things simple, Fraunhofer is working with stereo sound for the first generation. Second-generation immersive audio will bring object-based sound into VR, making the environments all-the-more realistic and believable.
Mary Spio and her company, Next Galaxy, are using a very different approach from Fraunhofer's. Next Galaxy uses sound tracing to imitate sound profiles for different environments. Spio said this is a physics-based approach of estimating the environment and replicating the sound to match it.
The company uses seven different variables to determine whether to lower or raise sound levels to mimic a sound's 3D origin. Spio said that by using this technology, even some compressed audio files can be processed this way, and that some existing audio content can benefit from sound tracing enhancements, as well.
The Trouble With Immersive Audio
Immersive, object-based audio will bring VR immersion to a whole new level, but that kind of audio profile will come with its cons. Developers and audio engineers are going to have to learn how to use the new set of tools required to create these immersive audio profiles. Spio believes that there are too many tools on the market right now. People might be waiting for a clear standard.
Jan Nordmann mentioned that immersive audio is going to cost more in processing resources. Stereo sound is the easiest to process and is much simpler than surround sound. Immersive audio will be a similar step up and will be even more demanding still.
It's Not Just The Audio Experts Talking About Audio
Later in the day we heard from three more speakers who reiterated several of the points that Nordmann and Spio brought up earlier. They all believe that sound is a key element in the VR experience.
Ricardo Laganaro, Director of O2 Films in Rio de Janeiro, has been working on a project to build "The Museum Of The Future," which focuses on current and future technology. He and his team are not VR developers -- they are film makers -- so they have been learning a lot as they go, but one thing they learned quickly is that audio can make or break an experience.
Laganaro and his team were trying to build an experience that would intentionally throw you off balance. They experimented with and without sound, and they found that having audio that tracked with the environment made it a much more powerful experience.
Resh Sidhu is the Creative Director at Framestore VR Studios. She and her team have been responsible for some incredibly immersive 4D experiences, such as the Game of Thrones: Ascend the Wall VR experience, which debuted at the second annual HBO Game of Thrones exhibition in 2014, and the Merrell virtual hike experience that had people walking a rope bridge in VR and in, er, literal reality at the same time.
One of the main points in Sidhu's speech, "The Power of Virtual Immersion," is that sound is an integral part of an immersive experience. She said that building the soundscape for a virtual experience should be considered from day one. Even the simplest sound should not be underestimated, because it can have a powerful effect on the experience itself. "VR is about fooling all the senses," Sidhu said.
The last presenter to mention immersive audio was Josh Farkas, CEO of Cubicle Ninjas. His take on developing experiences for VR is that audio and input are both just as important, if not more important than the visual experience. Without believable audio, it doesn't matter how good the visuals are, it won't be as great as it should be.
Farkas mentioned an experiment that the creators of Radial-G performed on pain management therapy with virtual reality. He didn't have the data on hand, but he said that their findings showed somewhere around a 20 percent improvement in effectiveness when audio was added.
Farkas's company has built a number of virtual reality experiences, and his team has learned that 3D positional audio makes a huge impact on the user experience. He recommends using multilayer audio with different sounds layered overtop of themselves and multiple points of audio. An example he used was of his company's Guided Meditation VR application, where you are sitting on a tropical beach. The sound coming from the ocean is constant, but there are other environmental sounds, such as seagulls flying overhead, that move about throughout the scene. 3D positional audio sources are key to make that work.
What About The Reverb?
Farkas did mention one thing that didn't get much attention by anyone else -- in fact, he specifically called out that no one talked about it. He believes that reverb is incredibly important when creating a sound environment. Applying the correct reverb can mean the difference between believing you are actually in the virtual environment, and thinking that something is slightly off, which can break the immersive feeling.
It's clear to see that audio is the next frontier for true immersion in virtual reality. Developers and audio professionals are starting to consider its impact on the experiences being created for VR, which is great, but it's also clear there is much more work to be done. While everyone agreed that audio is of massive importance, everyone seems to have their own approach to how this will be done. Eventually a standard will emerge, but I think we'll be seeing a fair bit more experimentation before that happens.
Update, 10/21/15, 2:17pm PT: Fixed typos.
Follow Kevin Carbotte @pumcypuhoy. Follow us @tomshardware, on Facebook and on Google+.
