MVF-GNN: Multi-View Fusion With GNN for 3D Semantic Segmentation

Abstract

Due to the high cost of obtaining 3D annotations and the accumulation of many 2D datasets with 2D semantic labels, deploying multi-view 2D images for 3D semantic segmentation has attracted widespread attention. Fusion of multi-view information requires establishing local-to-local as well as local-to-global dependencies among multiple views. However, previous methods that are based on 2D annotations supervision cannot model local-to-local and local-to-global dependencies simultaneously. In this letter, we propose a novel multi-view fusion framework with graph neural networks (MVF-GNN) for multi-view interaction and integration. First, a multi-view graph based on the associated pixels in multiple views is constructed. Then, a multi-scale multi-view graph attention network (MSMVGAT) module is introduced to perform graph reasoning on multi-view graphs at different scales. Finally, an attention multi-view graph aggregation (AMVGA) module is introduced to learn the importance of different views and integrate multi-view features. Experiments on the ScanNetv2 benchmark dataset show that our method outperforms state-of-the-art 2D/3D semantic segmentation methods based on 2D annotations supervision.