Projection mapping based on BRDF reconstruction from single RGBD image

Abstract

There have been many researches on projection mapping focused on target objects tracking, geometric shape recovering or virtual materials simulating such as clothes. However, few people pay attention to the material of target object which actually influences the visual results of projection. We present a new projection mapping framework based on BRDF reconstruction for the goal of more real projection results by enhancing the effects of Augmented Reality. In the framework, 3D computer vision method is used to reconstruct the BRDF of target object with a single RGBD image. A new algorithm is proposed using two Convolutional Neural Networks(CNN) which can predict both normal map and reflectance map of the target surface simultaneously with the RGBD image. The predicted maps are used to render the content to be projected onto the target object. Our BRDF reconstruction algorithm can recover several materials in one scene correctly in use of just one image. Experimental results show our framework has impressive performance and relatively accurate consequence.