Bidirectional Mask Selection for Zero-Shot Referring Image Segmentation

Abstract

Zero-shot referring image segmentation (RIS) aims to segment a referent mask via a natural language expression, without any training. Although existing research has made some progress, the lack of a training process in zero-shot learning results in insufficient information, leading to poor zero-shot segmentation performance. We propose a Bidirectional Mask Selection (BMS) framework, which is the first work to incorporate the negative masks into zero-shot RIS. Our idea is based on leveraging the negative masks’ semantic context information around target semantic to enhance the understanding of cross-modal fine-grained correlation. Further, we propose a novel mask adaptive fusion strategy to combine the complementary information from positive and negative masks without additional training. In the experiments, BMS has demonstrated outstanding performance on three prominent RIS datasets, and it has surpassed even the most advanced weakly supervised methods on the RefCOCOg datasets. Code will be available at https://github.com/pcc-99/BMS.