Physical prior-guided deep fusion network with shading cues for shape from polarization

Abstract

Shape from polarization (SfP) is a powerful passive three-dimensional imaging technique that enables the reconstruction of surface normal with dense textural details. However, existing deep learning-based SfP methods only focus on the polarization prior, which makes it difficult to accurately reconstruct targets with rich texture details under complicated scenes. Aiming to improve the reconstruction accuracy, we utilize the surface normal estimated from shading cues and the innovatively proposed specular confidence as shading prior to provide additional feature information. Furthermore, to efficiently combine the polarization and shading priors, a novel deep fusion network named SfPSNet is proposed for the information extraction and the reconstruction of surface normal. SfPSNet is implemented based on a dual-branch architecture to handle different physical priors. A feature correction module is specifically designed to mutually rectify the defects in channel-wise and spatial-wise dimensions, respectively. In addition, a feature fusion module is proposed to fuse the feature maps of polarization and shading priors based on an efficient cross-attention mechanism. Our experimental results show that the fusion of polarization and shading priors can significantly improve the reconstruction quality of surface normal, especially for objects or scenes illuminated by complex lighting sources. As a result, SfPSNet shows state-of-the-art performance compared with existing deep learning-based SfP methods benefiting from its efficiency in extracting and fusing information from different priors.