VirInteraction: Enhancing Virtual-LiDAR Points Interaction by Using Image Semantics and Density Estimation for 3D Object Detection

Distant object detection is a difficult problem in LiDAR-based 3D object detection. In recent years, the 3D detection of distant objects has achieved great success with the proposed fusion method of the virtual points generated by depth completion and LiDAR points. However, the inaccuracy of depth completion brings a lot of noise which significantly reduces the detection accuracy. To reduce noise and improve the detection accuracy of distant objects, we propose a solution called VirInteraction, which is a semantic-guided Virtual-LiDAR fusion method to enhance the interaction of virtual points and LiDAR points. Specifically, VirInteraction mainly includes three new designs: 1) Foreground-based adaptive Voxel Denoising (FgVD), 2) Semantic neighboring Sampling (Se-Sampling), and 3) Multi-scale Density-aware Cross Attention (MDC-Attention). FgVD uses Kernel Density Estimation (KDE) to adaptively denoise the foreground and background voxels. Se-Sampling completes the shape cues of distant objects using bidirectional sampling based on self-attention mechanism. Meanwhile, we built on these two designs and VirConvNet to develop a more robust VirInterNet as our virtual-point-based backbone. Finally, MDC-Attention elegantly aggregates the features of the images and points at the feature level according to the density distribution. Extensive experiments on KITTI and nuScenes datasets demonstrate the effectiveness of VirInteraction.