A Learning-Based Framework for Depth Perception using Dense Light Fields

Abstract

The rapid development of optical sensors technology has accompanied a growing demand for visual measurement systems in emerging areas that need to interpret the real three-dimensional physical world, such as self-driving cars, mobile robotics, Advanced Driver Assistance Systems (ADAS), and medical diagnostic in 3D imaging. In these systems, for modeling the physical world, it is necessary to unify visual information with depth measurements. Light Field cameras have the potential to be used in such systems as a versatile hypersensor. Since Light Fields represent the scene’s visual information from multiple viewpoints, it is possible to calculate the depth information through trigonometric operations. This paper proposes a learning-based framework that allows unifying scene depth with visual information obtained from Light Fields. The structure of the proposed framework is composed of four main modules. The deep learning modules consist of (i) a depth map estimation using a siamese convolutional neural network and (ii) an instance segmentation employing region-based convolutional neural network. The others two modules apply linear transformations: (iii) a module which applies the matrix transformations with camera intrinsic parameters to generated a new depth map of absolute distances and (iv) a module to return the distance of the selected objects. For the depth map estimation module this framework proposal a siamese neural network called EPINET-FAST, which allows for generating depth maps in less than half the time of the original EPINET. A case study is presented using Dense Light Fields captured by a Lytro Illum camera (plenotic 1.0). The case study seeks to exemplify the processing time of each module, allowing researchers to isolate critical points and propose changes in the future, seeking a processing that can be applied in real time.