How to Allow Interaction With Reality While in Virtual Reality

Augmented reality provides a live view of the real world with computer generated elements superimposed. Pilots have long used head-up displays to access air speed data and other parameters while they fly. Some smartphone cameras can superimpose computer-generated characters on to the view of the real world. And emerging technologies such as Google Glass aim to superimpose useful information on to a real world view, such as navigation directions and personal data.

But there’s a related problem that most people will not yet have considered. Imagine wearing a virtual reality headset and that you are immersed in a virtual world quite unlike the physical one around you. Now suppose you want to take a sip of water from a cup on the desk in front of you.

The only way to succeed is by feeling your way to the cup while still immersed in the virtual world or by removing the virtual reality headset and returning to the physical world. Neither of these is particularly good, say Pulkit Budhiraja and pals at the University of Illinois at Urbana-Champaign, who have come up with a solution.

These guys have been testing ways of superimposing physical reality onto a virtual reality experience. The goal is to find a way to allow users to interact with real physical objects while they remain immersed in a virtual world—a kind of augmented virtual reality

Budhiraja and co began by modifying an Oculus Rift virtual reality headset with a pair of cameras that produce a stereo view of the real world in front of the headset. They then came up with four different ways of superimposing the real world images onto the virtual world for the task of picking up and drinking from a cup, while remaining immersed.

link.