Efficient RGB-D Co-Salient Object Detection via Modality-Aware Prompting

Abstract

RGB-D co-salient object detection (Co-SOD) aims to identify and segment co-occurring salient objects in a set of correlated images and depth maps. Most existing RGB-D Co-SOD methods fully fine-tune the dual-stream encoder-decoder architecture and fuse the RGB and depth features using a complex feature fusion strategy, which is expensive to train owing to the large number of parameters that need to be updated during the feature extraction and fusion process. In addition, current methods do not pay sufficient attention to differentiate co- salient information from non-co- salient information effectively. This interfering information affects the localization of co-salient targets. Therefore, this study proposes a simple and effective modality-aware prompting network (MAPNet) for efficient RGB-D Co-SOD. MAPNet mainly performs RGB-D Co-SOD through two approaches, namely modal fusion and consensus feature extraction, using a multimodal prompt generator (MPG) and consensus feature extraction module (CFEM), respectively. Specifically, the MPG module guides the depth features in the fine-tuned backbone network from the RGB features obtained in the frozen backbone network for fusion in hyperbolic spaces to generate multilevel modal cues that are subsequently injected into the fine-tuned backbone network for efficient modal fusion. The CFEM uses RGB features to generate an image salient prior, combines the salient prior with the highest level of fusion features to obtain the central point, and uses the salient features closer to the central point as the consensus features of the image group. In addition, contrast loss is introduced to separate the synergistic and non-synergistic salient features to obtain pure co-salient features. The trained MAPNet delivered state-of-the-art performance on three benchmark datasets (RGB-D CoSal1k, RGB-D CoSal150, and RGB-D CoSeg183), with the structure-measure improved by 2.1% on the RGB-D CoSeg183 dataset. The codes are available at https://github.com/trumpetor/MAPNet. Note to Practitioners—This study presents a straightforward and effective modality-aware prompting network (MAPNet) designed for efficient RGB-D Co-SOD. Initially, the MPG module of MAPNet enables the effective integration of RGB and depth modalities through prompt learning. Subsequently, the CFEM employs the pixel group centroid proxy and top-k selection mechanism to extract high-level integrated features and salient prior consensus features, which serve as coordinated saliency features for image groups. Finally, the coordinated obtained salient and integrated features are input into the decoder to generate predictions.