Weakly supervised semantic segmentation for ancient architecture based on multiscale adaptive fusion and spectral clustering

Abstract

Existing methods of weakly supervised semantic segmentation for ancient architecture have several limitations including difficulty in capturing decorative details and achieving precise segmentation boundaries due to the many details and complex shapes of these structures. To mitigate the effect of the above issues in ancient architecture images, this paper proposes a method for weakly supervised semantic segmentation of ancient architecture based on multiscale adaptive fusion and spectral clustering. Specifically, low-level features are able to capture localized details in an image, which helps to identify small objects. In contrast, high-level features can capture the overall shape of an object, making them more effective in recognizing large objects. We use a gating mechanism to adaptively fuse high-level and low-level features in order to retain objects of different sizes. Additionally, by employing spectral clustering, pixels in ancient architectural images can be divided into different regions based on their feature similarities. These regions serve as processing units, providing precise boundaries for class activation map (CAM) and improving segmentation accuracy. Experimental results on the Ancient Architecture, Baroque Architecture, MS COCO 2014 and PASCAL VOC 2012 datasets show that the method outperforms the existing weakly supervised methods, achieving 46.9%, 55.8%, 69.9% and 38.3% in Mean Intersection Over Union (MIOU), respectively. The code is available at https://github.com/hao530/MASC.git