Collaborative 3D object detection by smart vehicles considering semantic information and agent heterogeneity

Abstract

Collaborative perception can significantly enhance perception performance through information sharing among multiple smart vehicles and roadside perception systems. Existing methods typically require sharing and aggregating all information from other collaborators, which can significantly reduce collaborative performance in detecting distant or occluded objects due to the large amount of redundant information in the final perception. To address this issue, we propose a novel collaborative framework, named the Semantic Aware Heterogeneous Network (SAHNet), which extracts, shares and fuses perceptually crucial and useful information among heterogeneous collaborators to improve the performance of 3D object detection. Specifically, we first design a Foreground and Boundary Feature Selection (FBFS) to enhance meaningful feature extraction. Then a Heterogeneous Feature Transfer module (HFF) is then proposed to account for collaborators’ heterogeneity to better transfer perception-critical features. Finally, we introduce a Semantic Feature Fusion module (SFF) that effectively aggregates features using semantic information. The proposed framework has been extensively compared and evaluated on two simulation datasets and one real-world dataset. The experimental results demonstrate that SAHNet consistently outperforms existing methods in collaborative object detection, demonstrating strong robustness even under conditions with localization noise and time delays. Additionally, we have provided a comprehensive ablation study to illustrate the effectiveness of each module within our framework.