Apply Masked-attention Mask Transformer to Instance Segmentation in Pathology Images

Abstract

Instance segmentation can be applied for the discrimination and diagnosis of cancer cells in pathology images. Accurate segmentation of each pathological cell in the pathology images can improve the efficiency of clinical diagnosis. In this paper, we aim to evaluate the state-of-the-art transformer-based instance segmentation method, masked-attention mask transformer (Mask2Former)[1], on pathology datasets. With the pretrained model of Mask2Former on the natural image instance segmentation dataset, we show that Mask2Former can be adaptive to small pathological datasets and achieve comparable or even better instance segmentation performance compared with the state-of-the-art task-specific pathology image instance segmentation methods.