Environment-Dependent Depth Enhancement with Multi-modal Sensor Fusion Learning

This paper presents a new learning based multimodal sensing paradigm within a probabilistic framework to improve the depth image measurements of an RGB-D camera. The proposed approach uses an RGB-D camera and laser range finder to provide an improved depth image using convolutional neural network (CNN) approximation within a probabilistic inference framework. Synchronized RGB-D and laser measurements are collected in an environment to train a model, which is then used for depth image accuracy improvements and sensor range extension. The model exploits additional RGB information, which contains depth cues, to enhance the accuracy of pixel level measurements. A computationally efficient implementation of the CNN allows the model to train while exploring an unknown area to provide improved depth image measurements. The approach yields depth images containing spatial information far beyond the suggested operational limits. We demonstrate a nearly three-fold depth range extension (3:5m to 10m) while maintaining similar camera accuracy at the maximum range. The mean absolute error is also reduced from the original depth image by a factor of six. The efficacy of this approach is demonstrated in an unstructured office space.