A Single Camera Image Based Approach for Glossy Reflections in Mixed Reality Applications

Proper scene inference provides the basis for a seamless integration of virtual objects into the real environment. While widely neglected in many AR/MR environments, previous approaches providing good results were based on rather complex setups, often involving mirrored balls, several HDR cameras, and fish eye lenses to achieve proper light probes. In this paper we present an approach requiring a single RGB-D camera image only for generating glossy reflections on virtual objects. Our approach is based on a partial 3D reconstruction of the real environment combined with a screen-space ray-tracing mechanism. We show that our approach allows for convincing reflections of the real environment as well as mutual reflections between virtual objects of an MR environment.