Learning better contrastive view from radiologist’s gaze

Abstract

Recent advancements in self-supervised contrastive learning have shown significant benefits from utilizing a Siamese network architecture, which focuses on reducing the distances between similar (positive) pairs of data. These methods often employ random data augmentations on input images, with the expectation that these augmented views of the same image will be recognized as similar and thus, positively paired. However, this approach of random augmentation may not fully consider the semantics of the image, potentially leading to a reduction in the quality of the augmented images for contrastive learning. This challenge is particularly pronounced in the domain of medical imaging, where disease-related anomalies can be subtle and easily corrupted. In this study, we initially show that for commonly used X-ray images, traditional augmentation techniques employed in contrastive pre-training can negatively impact the performance of subsequent diagnostic or classification tasks. To address this, we introduce a novel augmentation method, i.e., FocusContrast, to learn from radiologists’ gaze during diagnosis and generate contrastive views with guidance from radiologists’ visual attention. Specifically, we track the eye movements of radiologists to understand their visual attention while diagnosing X-ray images. This understanding allows the saliency prediction model to predict where a radiologist might focus when presented with a new image, guiding the attention-aware augmentation that maintains crucial details related to diseases. As a plug-and-play and module, FocusContrast can enhance the performance of contrastive learning frameworks like SimCLR, MoCo, and BYOL. Our results show consistent improvements on datasets of knee X-rays and digital mammography, demonstrating the effectiveness of incorporating radiological expertise into the augmentation process for contrastive learning in medical imaging.