It's 8:00am on a windy morning in Chatham, England, and Matt Daly is not pleased. The weather in the seaside U.K. town is living up to its dour reputation, and rain squalls and 50 mph blasts of wind off the wintry Atlantic have kept the small assembled crowd of videographers and journalists huddled under shelter. Daly is here to film a virtual reality movie about the HMS Cavalier, a massive 1942 destroyer hulking over the dockyards, and if the wind and rain don't let up, the drone can't fly, and if the drone can't fly, he loses one of his key scenes.
MORE: The History of Virtual Reality
MORE: The Key To Virtual Reality Immersion Is The Audio
MORE: The Past, Present, And Future Of VR And AR: The Pioneers Speak
MORE: All Virtual Reality ContentMORE: Virtual Reality In The Forums
Shooting In VR: A New Film-Making Frontier
Virtual reality content is new to many consumers, but it's just as new to producers. Video production has undergone massive shifts in the last decade, with new tools like cheap DSLR cameras and LED lighting dropping the entry costs dramatically, but the basics have remained the same until now.
VR is the wild west, with no established workflows or procedures to follow. It's an entirely new way to think about visual media, with challenges and considerations far beyond what you'd encounter on a typical shoot. Think for a second about a fundamental fact of 360-degree video: The minute you start the camera, the crew has to be out of sight. If you can see the camera, then you're on film, meaning there's no easy way for the director to watch scenes happen. Furthermore, the camera usually has to be static, because not only can movement cause nausea in viewers, but having something as basic as a live camera operator in the room presents a host of editing issues down the line.
This is the basic way a VR scene is shot: The camera is mounted to a narrow tripod, or hung from an overhead mount. The actors drill their lines and movements as if they're going to be on stage. Editing and cutting in 360° VR is disconcerting for audiences, so scenes have to play out in one take. The director preps the actors as best they can, and supervises the camera placement, but considering most GoPro-based systems can't stream a live feed, there's no good way to be sure that the camera or actors are positioned exactly right. Then someone yells "action," and everyone scatters out of the room, leaving the actors alone with the camera to perform their lines as best they can.
Daly and his Special Projects team at Wargaming have produced two previous VR movies, and in each one they were feeling their way as they went, experimenting with different tools and strategies. Their team started out using the Freedom 360, one of the most popular products on the market today. Like many current VR products, the Freedom 360 is based on arranging GoPro action cameras into a plastic rig that points them in a circle. In the case of the Freedom 360, it's six GoPros giving full spherical coverage (though other GoPro-based systems use as many as 14 cameras). This provides extremely high resolution capture (current GoPro models can capture 4K footage) but can be awkward to manage, and the rig can make accessing ports, batteries and SD cards a pain. It's also overkill, as most popular VR streaming platforms, including LittlStar and YouTube360 are limited to 4
Post processing is also a massive headache. GoPros are mostly automated cameras, and cameras on different sides of the rig may shift their exposure up or down depending on if they're near direct light, or even if someone walks nearby. White balance can also shift, and the end result is that straight out of the camera, each individual feed might look very different.
The general process is that each camera's footage has to be color-corrected and tweaked to match the others as best as possible, and then run through expensive software such as VideoStitch or Autopano to combine the different video files into a single file. "It's just slow and labor intensive," said Daly. In addition to being slow and finicky, stitching (the process of combining the multiple camera feeds into a single 360° video) still isn't perfect.
Daly explained one of the issues with misaligned stitches: "If you're filming human subjects, and you have multiple stitch lines, you have to deal with the problems of your subjects crossing those lines. That isn't actually a dealbreaker as far as I'm concerned, but more problematic is when you have your subjects reside between two stitch lines, which gives them a sort of Frankenstein look, which is a dealbreaker for immersion outside of anything but horror."
Stitched video frequently has noticeable seams in where the footage doesn't perfectly align. While software is improving, tiny errors in camera position and intrinsic variation in the lenses makes a perfect stitch almost impossible. The process is also getting faster, but rendering a stitched video from multiple sources can also be extremely time consuming. Just a year or two ago, it wouldn't be unusual for a few minutes of video to take several days to stitch and render. Even today, a computer can take hours to stitch a video.
Both of the early videos from Wargaming were centered around tanks. Looking at them now, the footage is compelling, but there are noticeable stitch lines and severe distortion whenever an object approaches the edge of the frame. Still, it's evident what big ventures these were, with video from inside the cockpit of a WWII tank, or from a camera rig strapped to the top of a tank on the move.
Gaming House Bona Fides
Wargaming, the company behind these elaborate productions, is best known for the hugely popular online game World of Tanks and the follow ups World of Warplanes and World of Warships (which launched September 2015). Although it's still a relatively unknown company in the U.S., Wargaming has grown from a single office in Minsk, Belarus, to more than 4,000 employees across the globe. World of Tanks has been an unprecedented success, with Wargaming claiming more than 120 million registered players. Of course, the company won't state how many of those players are active or paying, but market research firm SuperData estimated that World of Tanks is the world's most profitable free-to-play MMO (massively multiplayer online) game in terms of revenue per player.
Wargaming is poised to move into more established properties as well; along the way, it has also acquired Gas Powered Games (the studio behind the Dungeon Siege games) and the license rights to both the Total Annihilation and Master of Orion franchises. (A Master of Orion reboot went into Steam Early Access on February 26.)
Despite this diversification, the main focus of Wargaming is, and will continue to be, WWII-themed content. To that end, Wargaming has decided that it's in its best interest to generate public enthusiasm for a range of topics related to WWII history. Most of these ventures have taken the form of educational content produced by the Wargaming Special Projects team.
Some of the projects Wargaming has undertaken include the funding of tank restorations and the annual "tankfest" at the U.K.'s Bovington Tank Museum. Additionally, it financed the recovery of a rare Dornier-17 bomber from the English Channel and the creation of an augmented reality (AR) reconstruction of the plane.
VR seems like a natural extension for a company focused on simulation games. Gamers are likely to be the first voracious consumers of VR content, not only because of the hardware crossover with games, but also because of a familiarity with interacting with artificial environments.
"It's a natural extension of the historical video content we were already making," said Daly. "Our audience is just perfectly fit for this kind of content. They're gamers, they understand interactive multimedia, and they're super enthusiastic about the subject matter."
Wargaming seems to be banking on this fact, and the production at Chatham was remarkably complicated.
The Gear, And Shooting Blind No More
One major advancement has been the use of Ricoh Theta VR cameras as monitors to preview shots. The multi-camera setups make it hard to preview your shot. Imagine taking a picture without ever looking through the viewfinder, or with your phone without being able to see the screen. Furthermore, with the slow speed of stitching, it might be hours or days before you were able to see what you shot.
"From a content creation standpoint, it's like we just got our monitors. We've been flying blind on all our previous videos," said Daly. The director of the shoot, Luke Kneller, clearly appreciated the new Thetas, and he had the camera men check and recheck each shot by streaming an image to their smartphones.
In an attempt to move away from complicated multi camera rigs that require elaborate stitching, Wargaming filmed the HMS Cavalier piece on a custom camera rig developed by two companies. Back-Bone, a Canadian firm, makes modified GoPros by stripping off the entire front of the camera and replacing it with a reinforced metal place and lens mount. The new system, dubbed the "Ribcage," can support much heavier and more complicated lenses in addition to sporting a tripod mount. The other company, Entapano, is a Japanese startup that makes a line of "Entaniya" super-wide fisheye lenses compatible with the Ribcage's lens c-mount.
The largest Entaniya is a single lens that gives the GoPro a 280° field of view. This means that pointing straight up, the lens will capture everything except the space directly below it, and pointed at a subject, it will capture everything except the camera operator behind it. The innovation in this system is that as a single-lens design, it doesn't require any stitching at all. Because the GoPro is capable of 4k footage, the final product should have similar resolution to the final product of even a six-camera rig.
Despite these advantages, the single-lens system doesn't do true 360°, and the Wargaming crew still relied on a multi-camera rig for most of the scenes they filmed, though a massively simplified one. Another Back-Bone-and-Entapano combo uses a reinforced baseplate to mount two Ribcage-reinforced GoPros back-to-back. These cameras were equipped with 220° lenses, providing a true spherical view. This system does introduce faint stitch lines, but at least on sample footage they seem far less noticeable than those from a more complicated system, and with improved resolution and field of view over the single lens.
From a post-production perspective, Kneller estimated that the two-camera system cut their time processing the footage after a shoot in half. Although this is significant, Kneller also stressed that these days, the resources needed for stitching pale in comparison to the processing power needed for computer graphics and compositing, which traditionally have always been the most resource-intensive parts of film making.
The simplicity of these one- and two-lens systems makes them extremely forgiving to shoot with, but Daly thinks the larger rigs still have their place. Stitching will improve and speed up over time, and as long as you keep the original footage, you can try to correct errors down the line. The huge output resolution of a half-dozen cameras might also become more important. "Right now streams might be capped at 4k, but once the infrastructure and technology starts evolving so we can deliver the content at higher and higher resolutions, we can always go back and re-release projects recorded months or even years ago on the multi camera rigs at high quality."
Shooting Challenges
One area in which the Entaniya systems excel over larger rigs is in mobility. The two-camera system was used in several experiments that the Wargaming crew tried with moving the camera. Most early experiments in VR have used a static camera. The minute you start to move the camera, it creates problems for a number of reasons. First off, whoever is moving the camera will either be on film, or you'll need to cut them out, resulting in black space or a major stitch line. Second, those stitch lines become much more visible as the camera moves; and third, if the viewer is wearing a headset, movement can be disconcerting or nauseating.
The first experiment with movement involved the two-camera system and an operator carrying it. The camera was carried alongside the actors, and they were instructed to treat the camera like another person (i.e., maintaining a certain personal space, looking directly at the lens, speaking to it, etc.). The hope was that the effect would be to create the feeling of actually being taken on a tour by the presenters.
Back on the docks in the midst of a persistent winter drizzle, the second experiment with mobility is still struggling. The plan for the drone was to attach the two lens GoPro system to a DJI S900 drone and fly it over the ship. The drone would track alongside the presenters as they walked from bow to stern, and then pull up and back to show a birds-eye of the entire dockyard.
The S900 is a professional drone platform, capable of flying for over 15 minutes at a time while carrying nearly ten pounds, but even so, it's struggling in the high winds coming off the ocean.
The challenge is that the camera isn't mounted on a gimbal; it's merely attached to a long rod below the drone. This meant the drone operator had to manage an incredibly steady flight in the lousy English weather, or risk creating the VR equivalent of a jittery roller-coaster ride.
Gimbals are typically used to stabilize footage from drones, but there aren't really any effective gimbals currently designed for VR. A traditional gimbal would have put the camera so close to the drone's hull that it would feel like it was hovering oppressively just above your head. Additionally, there was concern that a gimbal, even though they are designed to stabilize footage, would actually introduce enough movement, especially of the rolling and tilting variety, to make the drone footage severely nauseating.
The risk of nausea seems low, but the team is continually worried about it. Certain people respond poorly to VR, and the last thing the Wargaming crew wants is to make their users sick. Tilting forward or backward isn't too bad, but the minute the scene starts rolling side to side, some viewers react with virtual seasickness as if they're on a ship in choppy water.
After several failed attempts, tensions were running high. This wasn't an easy shot to nail, as the drone's position and the actors walk along the ship had to sync up, and all in between gusts of wind. There were limited chances to try, too, with each attempt using up one of the seven batteries they had available for the shoot. At this point, it was also reported that the old veteran sailors who were going to load and fire the ship's main cannon with a blank had somehow managed to jam the mechanism, ruining a planned aerial shot of the gun firing as well.
On the fourth attempt, though, all the variables aligned. The actors hit their marks, the drone stayed smooth and stable as it tracked alongside them, and then it burst high into the air, tiny dangling GoPro in tow.
MORE: The History of Virtual Reality
MORE: The Key To Virtual Reality Immersion Is The Audio
MORE: The Past, Present, And Future Of VR And AR: The Pioneers Speak
MORE: All Virtual Reality Content
MORE: Virtual Reality In The Forums
Update, 4/2/16, 3:06pm PT: Fixed typos. Clarified 120 million players; the article originally stated that it was 100 million
