Remote sensing scene graph generation for improved retrieval based on spatial relationships

Abstract

RS scene graphs represent RS scenes as graphs with objects as nodes and their spatial relationships as edges, playing a crucial role in understanding and interpreting RS scenes at a higher level. However, existing RS scene graph generation methods, relying on deep learning models, face limitations due to their dependence on extensive relationship labels, restricted generation accuracy, and limited generalizability. To address these challenges, we proposed a spatial relationship computing model based on prior geographic information knowledge for RS scene graph generation. We refer to the RS scene graph generated using our method as SG-SSR for short. Furthermore, we investigated the application of SG-SSR in RS scene retrieval, demonstrating improved retrieval accuracy for spatial relationships between entities. The experiments show that our scene graph generation method does not rely on relationship labels, and has higher generation accuracy and greater universality. Moreover, the retrieval method based on SG-SSR outperformed other retrieval methods based on image feature vectors, with a retrieval accuracy index 0.098 higher than the alternatives(RemoteCLIP(mask)). The dataset and code are available at https://gitee.com/tangjiayitangjiayi/sg-ssr.