Generating counterfactual negative samples for image-text matching

The method of image-text matching typically employs hard triplet loss as its optimization objective to learn coarse correspondences based on object co-occurrence statistics. However, due to insufficiently sampled negative instances, this coarse correspondences not only leads to the model learning biases in semantic co-occurrence but also obscures the model’s understanding of crucial semantic and significant semantic contextual dependencies. In this study, we propose the Generating Feature-level and Relation-level Counterfactual Negative Samples method (GFRN) for image-text matching. This method utilizes prior knowledge and gradients to mask key regions or words to generate feature-level counterfactual negative samples, or disrupts their important contextual dependencies through Bernoulli distributions and self-supervised learning to generate relation-level counterfactual negative samples with sufficient information. Subsequently, we employ these counterfactual samples to construct contrastive triplet losses to enhance the training of the image-text matching model. Consequently, the model’s ability to understand crucial semantic concepts and complex dependency relationships is significantly enhanced, and semantic biases are greatly reduced. Compared to state-of-the-art methods, the proposed GFRN improves rSum by 3.9% on Flickr30K, 2.0% on MSCOCO1K, and 4.8% on MSCOCO5K, with significant improvements in R@1 across all datasets.