Multi-representation fusion learning for weakly supervised semantic segmentation

Abstract

Weakly Supervised Semantic Segmentation (WSSS) with image-level labels offers a promising solution to the expensive problem of pixel-level annotation. However, the prevalent use of Class Activation Maps (CAMs), while effective, often results in inaccurate object boundaries and poor edge details in generated pseudo-labels. To overcome these limitations, this paper presents a novel Multi-representation Fusion Learning (MFL) framework that leverages the remarkable capabilities of the Segment Anything Model (SAM) to enhance feature learning in WSSS. The MFL framework directly addresses the shortcomings of CAM-based pseudo-labels by incorporating rich semantic and edge information extracted from SAM. This is achieved through two dedicated modules: the Semantic-Guided Distilled Attention (SGDA) module and the Edge-Guided Distilled Attention (EGDA) module. These modules enable the network to learn more discriminative features by leveraging the SAM’s knowledge, leading to higher-quality pseudo-labels. Furthermore, the proposed Multi-representation Fusion Module (MFM), based on a dual-layer routing attention mechanism, effectively fuses the semantic and edge features learned by the SGDA and EGDA, resulting in more refined pseudo-labels for training. Extensive experiments on PASCAL VOC 2012 and MS COCO 2014 datasets demonstrate that the MFL framework significantly outperforms existing WSSS methods, achieving state-of-the-art performance.