FRORS: An Effective Fine-Grained Retrieval Framework for Optical Remote Sensing Images

Abstract

Fine-grained retrieval of remote sensing images is an image interpretation task that is still in its infancy. With the rapid development of convolutional neural networks (CNN) in the field of remote sensing, it has become possible for remote sensing image retrieval tasks to move toward more fine-grained classes. However, since current methods focus on how to construct similarity metrics between sample pairs, the model ignores the learning of fine-grained intraclass heterogeneity and interclass commonality features, which poses a huge challenge to fine-grained retrieval. To solve this problem, we propose a novel fine-grained retrieval framework of optical remote sensing (FRORS) images, which aims to improve fine-grained retrieval capabilities by constructing interaction and matching between intraclass heterogeneity features, interclass commonality features, and image features. Specifically, we first construct a fine-grained prototype memory (FPM) module, and continuously update the local prototype storage unit through a lightweight CNN to achieve a refined representation of fine-grained heterogeneity features. Furthermore, to learn interclass commonality, we propose a gram learning (GraL) strategy, which is achieved by learning the correlation between feature dimensions. On this basis, we introduce a gram-based metric match (GMM) mechanism, which fuses the prototype features representing intraclass heterogeneity and the gram vector representing interclass commonality through an embedding manner, thereby achieving the purpose of fully interactive matching between image features and fine-grained class features. With FPM, GraL, and GMM, our FRORS can better learn deep features representing fine-grained classes and promote the improvement of the network's fine-grained retrieval ability. Extensive experiments conducted on a self-constructed THUFG-OPT dataset prove that the proposed FRORS achieves state-of-the-art fine-grained retrieval performance, which is 5.75% higher than the baseline method on $\mathrm{mAP@10}$.