CMFFN: An efficient cross-modal feature fusion network for semantic segmentation

Abstract

Multimodal information fusion can improve the accuracy and robustness of semantic segmentation results. However, the differences between modalities, the complementarity of the information provided, and the effectiveness of the fusion process await further exploration. In addition, another modality significantly increases the computational complexity, parameters and model training cost. To address these issues, we propose a novel end-to-end multimodal dual-stream semantic segmentation network, called cross-modal filtering fusion network (CMFFN), which efficiently fuses features from RGB and raw point cloud modalities. In particular, our CMFFN does not require mapping raw point clouds into image formats or additional modality alignment designs. For the point cloud branch of CMFFN, to alleviate the problem of increasing computational resources with the number of input point clouds, we propose a lightweight backbone network based on a sparse query attention mechanism, which achieves a balance between feature extraction performance and training resource utilization on the ModelNet40. For multimodal fusion, we introduce the cross-modal scoring feature selection and fusion module (CMSSF) with learnable parameters, which provides a general approach to reduce expensive costs caused by modalities interaction. Benefiting from the effective pruning and fine-grained framework, CMFFN achieves up to 67.39% in mIoU with a 3. 64% gain compared to the latest state-of-the-art model CMNeXt on KITTI-360 raw data, while reducing parameters and computational workload by 14.4% and 15.6%, respectively. Furthermore, since CMFFN directly models raw 3D point cloud data, it produces visually higher quality masks than the ground truth in certain aspects, such as locating relative spatial positions of targets and predicting distant objects.