TLR-3DRN: Unsupervised single-view reconstruction via tri-layer renderer

Abstract

Single-view three-dimensional (3D) reconstruction is a challenging task in computer vision, focusing on reconstructing 3D objects from a single image. Existing single-view object reconstruction approaches typically rely on viewpoints, silhouettes, multiple views of the same instance, and strategy-specific priors, which are difficult to obtain in the wild. To address this issue, we propose a novel end-to-end single-view reconstruction method based on a tri-layer renderer, named the Tri-Layer Renderer-based 3D Reconstruction Network (TLR-3DRN). TLR-3DRN recovers 3D structures from original image collections without requiring additional supervision, assumptions, or priors. In particular, TLR-3DRN employs a tri-layer renderer that enables the model to extract more 3D details from unprocessed image data. To obtain an optimizable interlayer, we developed a robust interlayer generation network based on a nonparametric memory bank. Notably, we designed a joint optimization strategy for the overall framework. Additionally, a shape and texture consistency loss based on image–text models is proposed to enhance the optimization process. Owing to the aforementioned proposed modules, TLR-3DRN can achieve high-quality, diverse-category reconstruction under completely unsupervised conditions. TLR-3DRN is validated on synthetic datasets and real-world datasets. Experimental results demonstrate that TLR-3DRN outperforms state-of-the-art unsupervised and two-dimensional supervised methods, achieving performance comparable to 3D supervised methods.