Fusing the real and the virtual: A depth-camera based approach to Mixed Reality

Abstract

The seamless integration of the real and the virtual content is the ultimate yet unreached goal of Mixed Reality applications. Among others it requires mutual blocking and lighting between real and virtual objects. In this paper we present our approach of applying a low-cost depth camera, such as Kinect, allowing for an easy acquisition of depth images. However, as the quality of the raw input data is insufficient for this purpose, we apply a series of filter and optimization operations. This allows us to realize mutual real-time lighting and rigid interaction in a dynamic environment. Our approach produces an acceptable quality of images of low-frequency scenes at interactive frame rates on an off-the-shelf desktop computer.