The second generation of VR technology relies on a new, innovative “light field” projection. (The first commercial light field units are the HoloLens made by Microsoft and Magic Leap funded by Google.) In this design the VR is projected onto a semi-transparent visor much like a holograph. This permits the projected “reality” to overlay the reality you see normally without goggles. You could be standing in your kitchen and see the robot R2-D2 right before you in perfect resolution. You could walk around it, get closer, even move it to inspect it, and it would retain its authenticity. This overlay is called augmented reality (AR). Because the artificial part is added to your ordinary view of the world, your eyes are focused deeper than they are on a screen near your eyes, so this technological illusion is packed with presence. You almost swear it is really there.
Microsoft’s vision for light field AR is to build the office of the future. Instead of workers sitting in a cubicle in front of a wall of monitor screens, they sit in an open office wearing HoloLenses and see a huge wall of virtual screens around them. Or they click to be teleported to a 3-D conference room with a dozen coworkers who live in different cities. Or they click to a training room where an instructor will walk them though a first-aid class, guiding their avatars through the proper procedures. “See this? Now you do it.” In most ways, the AR class will be superior to a real-world class.
The reason why cinematic realism is advancing faster in VR than in cinema itself is due to a neat trick performed by head-mounted displays. To fill a gigantic IMAX cinema screen with the proper resolution and brightness to convince you it is a mere window into reality requires a massive amount of computation and luminosity. To fill a 60-inch flat screen with the same window-clear realism is a smaller challenge, but still daunting. It is much easier to get a tiny visor in front of your face up to that quality. Because a head-mounted display follows your gaze no matter where you look—it is always in front of your eyes—you see full realism all the time. Therefore if you make fully 3-D clear-as-a-window vision and keep it in view no matter where you look, you can create a virtual IMAX inside of the VR. Turn your gaze anywhere on the screen and the realism follows your gaze because the tech is physically attached to your face. In fact, the entire 360-degree virtual world appears in the same ultimate resolution as what’s in front of your eyes. And since what is in front of your eyes is just a small surface area, it is much easier and cheaper to magnify small improvements in quality. This tiny little area can invoke a huge disruptive presence.
But while “presence” will sell it, VR’s enduring benefits spring from its interactivity. It is unclear how comfortable, or uncomfortable, we’ll be with the encumbrances of VR gear. Even the streamlined Google Glass (which I also tried), a very mild AR display not much bigger than sunglasses, seemed too much trouble for most people in its first version. Presence will draw users in, but it is the interactivity quotient of VR that will keep it going. Interacting in all degrees will spread out to the rest of the technological world.
• • •
About 10 years ago, Second Life was a fashionable destination on the internet. Members of Second Life created full-body avatars in a simulated world that mirrored “first life.” A lot of their time was spent remaking their avatars into beautiful people with glamorous clothes and socializing with other members’ incredibly beautiful avatars. Members devoted lifetimes to building super beautiful homes and slick bars and discos. The environment and avatars were created in full 3-D, but due to technological constraints, members could only view the world in flat 2-D on their desktop screens. (Second Life is rebooting itself as a 3-D world in 2016, code-named Project Sansa.) Avatars communicated via text balloons floating over their heads, typed by owners. It was like walking around in a comic book. This clunky interface held back any deep sense of presence. The main attraction of Second Life was the completely open space for constructing a quasi-3-D environment. Your avatar walked onto an empty plain, like the blank field at a Burning Man festival, and could begin constructing the coolest and most outrageous buildings, rooms, or wilderness places. Physics didn’t matter, materials were free, anything was possible. But it took many hours to master the arcane 3-D tools. In 2009 a game company in Sweden, Minecraft, launched a similar construction world in quasi-3-D, but employed idiot-easy building blocks stacked like giant Legos. No learning was necessary. Many would-be builders migrated to Minecraft.
Second Life’s success had risen on the ability of kindred creative spirits to socialize, but when the social mojo moved to the mobile world, no phones had enough computing power to handle Second Life’s sophisticated 3-D, so the biggest audiences moved on. Even more headed to Minecraft, whose crude low-res pixelation allowed it to run on phones. Millions of members are still loyal to Second Life, and today at any hour about 50,000 avatars are simultaneously roaming the imaginary 3-D worlds built by users. Half of them are there for virtual sex, which relies more on the social component than on realism. A few years ago the founder of Second Life, Phil Rosedale, started another VR-ish company trying to harness the social opportunities of an open simulated world and to invent a more convincing VR.
Recently I visited the offices of Rosedale’s startup, High Fidelity. As the name implies, the aim of its project is to raise the realism in virtual worlds occupied by thousands—maybe tens of thousands—of avatars at once. Create a realistic thriving virtual city. Jaron Lanier’s pioneering VR permitted two occupants at once, and the thing I noticed (and everyone else who visited) was that other people in VR were far more interesting than other things. Experimenting again in 2015, I found the best demos of synthetic worlds are ones that trigger a deep presence not with the most pixels per inch, but with the most engagement of other people. To that end, High Fidelity is exploiting a neat trick. Taking advantage of the tracking abilities of cheap sensors, it can mirror the direction of your gaze in both worlds. Not just where you turn your head, but where you turn your eyes. Nano-small cameras buried inside the headset look back at your real eyes and transfer your exact gaze onto your avatar. That means that if someone is talking to your avatar, their eyes are staring at your eyes, and yours at theirs. Even if you move, requiring them to rotate their head, their eyes continue to lock onto yours. This eye contact is immensely magnetic. It stirs intimacy and radiates a felt presence.
Nicholas Negroponte, head of MIT’s Media Lab, once quipped in the 1990s that the urinal in the men’s restroom was smarter than his computer because it knew he was there and would flush when he left, while his computer had no idea he was sitting in front of it all day. That is still kind of true today. Laptops and even tablets and phones are largely ignorant of their owners’ use of them. That is starting to change with cheap eye tracking mechanisms like the one in the VR headsets. The newest Samsung Galaxy phone contains eye tracking technology so the phone knows precisely where on the screen you are looking. Gaze tracking can be used in many ways. It can speed up screen navigation since you often look at something before your finger or mouse moves to confirm it. Also, by measuring the duration of thousands of people’s gazes on a screen, software can generate maps that rank areas of greater or lesser attention. Website owners can then discern what part of their front page people actually look at and what parts are glanced over, and use that information to improve the design. An app maker can use gaze patterns of visitors to find which parts of an app’s interface demand too much attention, suggesting a difficulty that needs to be fixed. Mounted in a dashboard in a car, the same gaze technology can detect when drivers are drowsy or distracted.