SwinSAM: Fine-grained polyp segmentation in colonoscopy images via segment anything model integrated with a Swin Transformer decoder

Abstract

Polyp segmentation in colonoscopy imagery is a critical procedure in the early detection and preemptive management of colorectal cancer. In facilitating the diagnostic procedures, it is pivotal to attain segmentation with high precision, emphasizing fine-grained details which can potentially harbor crucial information regarding the disease state. To address the prevailing demand for more refined segmentation techniques, this study introduces an innovative framework “SwinSAM”, which ingeniously integrates a Swin Transformer decoder with a SAM encoder. The SAM model has seen over a billion images and possesses a strong capability for image comprehension. However, its training data primarily originates from natural images rather than medical ones. Hence, we designed an adapter module to infuse specific medical domain information into SAM. Furthermore, due to the varying sizes and shapes of polyps, along with their high blending degree with the background, the simplistic convolutional decoder in the original SAM model struggles to accurately segment the intricate details of polyps. This prompted us to utilize the Swin Transformer as the decoder. Additionally, considering the significant shape variations of polyps, we employed a multi-scale perception fusion module to process the deep features extracted by SAM. By using convolutions with different receptive fields, we can extract information about polyps of various shapes. Finally, we optimized the network parameters through multi-level supervision. Comprehensive experiments were conducted on five commonly used polyp segmentation datasets. The results validate that our proposed method achieves good performance across datasets with different polyp backgrounds.