Did you know that the first stereoscopic image viewer, the Stereoscope, was invented in the mid-1800s? What would you say if I told you the first flight simulator was built in 1928? Or that the patent for the first stereoscopic television, which looked an awful lot like the Google Cardboard viewers that we see today, was filed in 1960? Surely you’d be surprised if I told you that the first VR system, The Sword of Damocles, was built in 1968?
I didn’t know any of these facts until I started reading Jason Jerald’s incredible tome of VR information, aptly titled The VR Book. The wealth of virtual reality knowledge found in this book is something anyone working in the VR industry could really take advantage of. As the old adage goes, if we don’t learn from the past, we’re doomed to repeat it. Jerald has been in the industry for over twenty years, and he’s learned a thing or two in his career, and reading his book might just save you some pain if you’re creating experiences for VR.
Last week I had a chance to sit down with Jerald, and we talked about all things virtual reality.
A Little Bit About The Man
Jerald first became interested in technology in the 1980s when he “bribed his parents” to get him an Atari video game system. From that point on, he knew he wanted to be working with technology in one way or another, but it wasn’t until 1995 or 1996 when he had his first experiences with virtual reality:
“One of the first VR demos I saw [was] back in the 90s. It was this big LEGOLAND world. It was done in partnership with LEGO, Silicon Graphics and a company called Multigen Paradigm. They were laying LEGO bricks down, and they could scale themselves down, make themselves smaller to manipulate the smaller LEGOs. That just blew me away. Like, ‘Oh my God, I can’t believe this is happening!’”
Those first experiences with VR, over 20 years ago, changed his life. He instantly saw the power in the medium. His passion for the possibilities that the tech offered steered him down a career path working in virtual reality research. For the last 20 years, Jerald has been pursuing that passion, and he’s not afraid to admit that the topic excites him so much he could talk all day about it.
Jerald holds a Bachelor of Science with an emphasis in Computer Graphics, as well as and minors in Mathematics and Electrical Engineering, from Washington State University. He did his PhD in Computer Science at the University of North Carolina, where he focused on virtual reality and latency perception.
Currently, Jerald is the principal consultant of NextGen Interactions, a VR consultancy firm that he co-founded. His expertise has been called upon by some of the biggest companies in the world. His client list includes familiar names in the VR industry, including Valve and Oculus, as well as some major corporations in the aerospace industry, such as Raytheon and Lockheed Martin, and several universities.
The VR Book
Last year, Jerald decided to pour his wealth of knowledge into a book that anyone with an interest in working within the virtual reality industry could take advantage of. The VR Book includes information about the history of VR and the technologies that led up to the current age. It also has vast amounts of insight into what works (and doesn't) in VR. The premise, in his own words:
“This book focuses on human-centered design, a design philosophy that puts human needs, capabilities, and behavior first, then designs to accommodate those needs, capabilities, and ways of behaving. More specifically, this book focuses on the human elements of VR—how users perceive and intuitively interact with various forms of reality, causes of VR sickness, creating content that is pleasing and useful, and how to design and iterate upon effective VR applications.”
In the first pages of the book, he detailed how you should approach reading it if you are a newcomer to VR, a teacher, a student, a practitioner, or a VR expert. Different chapters are for different types of readers, but in general, The VR Book is for everyone interested in the technology. However, he told me that more than anything, it's written for content creators, "Someone that is looking to innovate and go beyond what everyone else is doing."
Now Is The Time
Our most burning question for Jerald was about the viability of consumer VR succeeding this time around.
Jason Jerald: Why is it different now? Back then [when he first got into VR] there were so many challenges. Now it’s the perfect storm for VR. Lower cost. Development tools. Hardware advancements. You don’t need a computer scientist anymore. Anyone has the opportunity to make the next killer app. A high school student has just as much chance to make it as a big company. He has advantages and disadvantages that the big company may not have.
There’s a very diverse pool of creators for VR content. VR is a whole new paradigm.
I’ve seen a lot of technical experts who played around with VR in the 90s, and those are the people that are the most pessimistic about it, often, because they had that tough experience back in the 90s. In my mind, one of the most common mistakes [people make] is that assuming that previous failures equal future failures. But no, this is completely different. Every technology has gone through failures, right? I can’t think of any technology that worked on the first iteration. A lot of times they failed miserably on the first attempt.
Tom's Hardware:What was the turning point for you? When did you know that this was viable as a consumer product, and not just something we’re trying to make work?
JJ:I remember around 2011, I was talking to the CEO of a company I was working with, and they were asking about head-mounted displays. And I just kind of laughed. I said, "We can’t even do this in a research lab, how are we supposed to do this in people's living rooms?" And that goes to show that even though I’m a so-called expert, that doesn’t mean I know all the answers.
It wasn’t someone from the research community that brought VR back; it was Palmer Luckey. I think he was 18 years old at the time.
And I think that’s one of the mistakes that a lot of VR researchers make--that they know it all. We certainly don’t know it all. We know a lot. We’ve been doing this for a long time, so I do disagree with the people that say we don’t know anything about VR. We do know some things, but there’s a lot more that we don’t know, compared to what we do know.
TH: Would you say that we are at the point where we’ve discovered enough about VR to say that we’re “past the crest of the hill” and on our way down figuring out the last bits, or are we still “climbing that hill,” figuring out how to make VR work properly?
JJ: I think we’re definitely still climbing that hill, and I think that’s one of the reasons that makes this [time] so exciting. There’s so much room to be creative. If we knew all the answers it wouldn’t be so exciting, and I see that continuing for a long time, if not forever. Because we haven’t figured out how the real world works. People are always arguing different theories of psychology--what makes the best products, all those sorts of things.
That’s for one world, and here we are saying, in virtual reality, that we’re sitting here creating multiple universes. To think that we’re going to figure that out anytime soon, I just don’t think that’s the case. I think it’s always going to be exciting, because there’s always going to be new crazy worlds to figure out. That’s why I say it’s not just an exciting time now. It’s always going to be.
TH: Do you see there being a time within our lifetime that we’ll see the "holodeck?"
JJ: It really depends how you define that. A lot of people ask, "What about photorealism?" And you know, the visual element of VR is absolutely important. You can definitely argue it’s the most important, but it’s not the only thing.
For example, resolution on displays have a long way to go. The human eye, under ideal conditions, can see one or two arcseconds of resolution. If you multiply that out over a 180-degree view, that’s a big number -- hundreds of thousands of pixels.
I’m not saying we need to do that, because we can do things like foveated rendering. That’s under very ideal conditions, and it’s not going to happen in most cases, but the point is, there’s still a long way to go.
And of course there’s more than just the visual element. Auditory is sort of the next obvious one. A lot of people just add that on as an afterthought, but really, you want to design your environment around it. You should think of that from the beginning. If you look at the best movies, the sound track is just amazing even...without visuals.
Going beyond that, [there is] the state of hand controls that are just starting to come out. [You have this] very subtle sense of where your body parts are located. I can reach out and touch my nose with my eyes closed, right? That’s not something we think about normally in the real world, but if we have that wrong in VR, then it just doesn’t seem right. Or, if your sense of balance or acceleration with your inner ear, your vestibular system, doesn’t match what you’re seeing, then suddenly we start to notice that, and you start getting sick. Normally we just take for granted that it works.
Then you can go beyond that, you can go to haptics, the sense of touch. We certainly haven’t figured that out, although we’re doing some basics. In research labs, they’re doing taste and smell. As far as a consumer product (laughs,) that, I think, is quite a way off.
TH: Have you tried the current VR units that are coming out in the next month?
JJ: Oh yeah! I have my Vive here. It’s just amazing.
A lot of people are concerned about content. The content is certainly there, but [in terms of] how to make these really compelling experiences, I think we have a way to go for that. Some of these experiences are amazing for the first few minutes -- it's like, "Oh my God! This is life changing!" But then it’s like, "Okay, now what do I do?" There’s not a lot of content out there yet that you want to stay in for longer than five of 10 minutes.
That’s a huge challenge that we need to overcome -- all of us, not just individuals. We need to all succeed together for people to take this seriously to amass diverse content.
TH: That’s the one criticism that outsiders seem to be getting right. The content isn’t there right now. If I spend $800 on a VR system and play the games for 10 minutes and never touch them again, that’s going to be a problem. So, given the state of the content right now, do you think we’re a bit premature for the hardware, or do you think this is the right time?
JJ: I think it’s the right time, and there are actually reasons for [the high cost of the hardware] beyond just the obvious money implications. If these high-end VR systems were available for say, $50, these companies just wouldn’t be able to deliver that quickly. They have to ramp up from something small that we’re still debugging [and we're] still figuring out how to do that hardware well at the consumer level. If it was at a lower price, from a manufacturing point of view, and a distributors point of view, you wouldn’t be able to get them out there that quickly. So, in some ways, that high cost results in more reasonable shipments of the hardware, and then they ramp it up more slowly.
Everyone is saying how slowly it’s coming. They want it here now. That’s one of the challenges with VR right now. You go in these five minute experiences and are totally blown away, and then once you realize what the potential is, people’s minds just go crazy. They start demanding these crazy things, and then, you know, we can’t deliver it fast enough.
TH: A lot of people are expecting the 10-hour, story-driven games, and we’re not there yet. Right now we’re looking at short arcade, five-minute, 10-minute bursts. Something that you can come in and play for a bit.
JJ: There's a ton of factors that contribute to sickness, or injury, or whatever it might be, but that’s one of the major factors: how long you’re in there. Even [in] a really well-optimized experience that minimizes issues, you eventually deal with something like eye strain, or headaches. And so, I think it is going to start with those really small, five- or 10-minute experiences until people get accustomed to those really long experiences.
So in some ways it's good that a lot of these demos are short. If someone is in there for four or five hours... some people can do that, but for a lot of people, that’s just not feasible. Designing these short experiences is a good way to start, even though it's limiting in some ways.
That five-minute demo that everyone is seeing now by going to a conference, or trade show, or whatever, is very different than owning one in your home and playing a game every evening. How people receive that is still very much unknown at this point.
One of the things that partly solves the content problem is user-created content -- for example, Tilt Brush, where you’re creating your own thing that’s different every time, or seeing what other people are doing.
Or Content creation. I’m working with a company now, that we’re looking [into] basically doing CAD operations; building worlds around you, that can be brought into other applications itself.
But also, there’s the social aspect that everyone is talking about. If you think about the content being the people, you can have some basic objects in the world. You can go in there, and that other person is unpredictable. He’s creating content on the fly, just by doing his motions. My understanding is, people go in [a VR environment] for hours just to hang out, and joke around and such. Which is, you know, a bit of a surprise.
TH: So what do you think of the current Vive and the Touch controllers. Are they good examples of a first-generation hand controller?
JJ: Oh absolutely! To be honest, I was a little bit disappointed with the head mounted displays, not because they are head mounted displays, but because, what are you going to do? You don’t have your body in this world. You’re this disembodied point in space, this observation point. If you look down, you don’t see your hands, you don’t see your body, any of that. So in my mind, interacting with the world is what VR is about.
You could argue, if you want to go to extremes, [that] it’s not VR if it’s not interactive, if you’re only looking around. And so, I’m just so happy to see that hands are being brought into the consumer space so quickly.
But it depends on your definition of first generation.
We’ve been doing this for a long time in the research world. We call them wands. We just had one hand, you just wouldn’t do VR research if you didn’t have a way to interact with the system. You wouldn’t have considered doing it. You’ve got to interact with the world in some way.
So that’s been around for a while. And then there’s the Razer Hydra, which unfortunately didn’t get much traction, it didn’t take off the way everyone would have liked to see, and there’s the Sony Playstation Move. So essentially, they [were] doing that years ago, but the difference, of course, is in a head mounted display, suddenly the hands become so much more important, so much more natural. When you’re looking at a screen, motion controls are kind of cool, but VR is just perfect for motion control.
TH: What do you think about Oculus’ move of releasing the headset months before the Touch controllers? A lot of people would think it’s an incomplete package without the full controllers, and I get the impression you are in that camp.
JJ: I certainly love having the hands in there, so it’s definitely not ideal, but there’s other decisions other than design. Like, in my book it’s all about design, but it doesn’t look at those other factors, like market factors, and financial factors. And then there’s the question of people creating content, there’s almost a divide. Some people have controllers, and some people don’t.
You could say “Well, let’s support both,” but those are such completely different experiences that you almost have to design the game around those interactions. If you design for both, and sort of take the average, then it’s not ideal either.
And of course, there’s more than just wand controllers -- everything from treadmills, to microphones with voice recognition, to eye tracking, and crazy custom hardware that people won’t likely have in their homes, but it’s great for location-based entertainment. I mean, it’s just wide open with what’s possible. It’s such a wide design space.
At least with the hardware that is shipping, even though it is split like we said, at least it narrows it down to a few types of input devices, instead of everyone trying to create customized, strange solutions with hacked-together hardware that just doesn’t work for consumers.
That’s the area that I’m so excited about, because in the research world, not many people were doing two-handed interactions like the consumer space is pushing now. A lot of VR applications were only using one hand, and there’s just so much more you can do with two hands, right? In the real world, we think we use our dominant hand, but our left hand does a lot of things. It provides stability, it provides a reference frame if you’re holding a notebook, or you’re writing on a piece of paper, or you’re peeling a potato. If you’re holding that in that hand, that suddenly becomes easy and natural to do, vs one hand
Human Centered Design of Immersion Interactions
Next week, during VRDC in San Francisco, Jerald will be giving a talk about the interaction of virtual reality that will discuss how to design interactions using these new hand controllers, and how to interact with your hands in VR, and "design patterns." Examples of interaction patterns include walking patterns, which Jerald said can be further broken down into walking techniques such as redirected walking, treadmills and walking in place; and 3D multi-touch patterns, such as those demonstrated in the recent preview of the Unreal Engine VR editor.
Jerald’s talk will take place on Tuesday, March 15 at 4:40pm EST in Room 2002 of the West Hall during VRDC. Following the 30-minute lecture, Jerald will be signing copies of The VR Book in the bookstore in the lobby above the Exhibit Hall. If you're attending, go find him and pick his brain.
Follow Kevin Carbotte @pumcypuhoy. Follow us on Facebook, Google+, RSS, Twitter and YouTube
