Accurate and complete neural implicit surface reconstruction in street scenes using images and LiDAR point clouds

Abstract

Surface reconstruction in street scenes is a critical task in computer vision and photogrammetry, with images and LiDAR point clouds being commonly used data sources. However, image-only reconstruction faces challenges such as lighting variations, weak textures, and sparse viewpoints, while LiDAR-only methods suffer from issues like sparse and noisy LiDAR point clouds. Effectively integrating these two modalities to leverage their complementary strengths remains an open problem. Inspired by recent advances in neural implicit representations, we propose a novel street-level neural implicit surface reconstruction approach that incorporates images and LiDAR point clouds into a unified framework for joint optimization. Three key components make our approach achieve state-of-the-art (SOTA) reconstruction performance with high accuracy and completeness in street scenes. First, we introduce an adaptive photometric constraint weighting method to mitigate the impacts of lighting variations and weak textures on reconstruction. Second, a new B-spline-based hierarchical hash encoder is proposed to ensure the continuity of gradient-derived normals and further to reduce the noise from images and LiDAR point clouds. Third, we implement effective signed distance field (SDF) constraints in a spatial hash grid allocated in near-surface space to fully exploit the geometric information provided by LiDAR point clouds. Additionally, we present two street-level datasets—one virtual and one real-world—offering a comprehensive set of resources that existing public datasets lack. Experimental results demonstrate the superior performance of our method. Compared to the SOTA image-LiDAR combined neural implicit method, namely StreetSurf, ours significantly improves the F-score by approximately 7 percentage points. Our code and data are available at https://github.com/SCH1001/StreetRecon.