LuoJiaHOG: A hierarchy oriented geo-aware image caption dataset for remote sensing image–text retrieval

Abstract

Image–text retrieval (ITR) is crucial for making informed decisions in various remote sensing (RS) applications, including urban development and disaster prevention. However, creating ITR datasets that combine vision and language modalities requires extensive geo-spatial sampling, diverse categories, and detailed descriptions. To address these needs, we introduce the LuojiaHOG dataset, which is geospatially aware, label-extension-friendly, and features comprehensive captions. LuojiaHOG incorporates hierarchical spatial sampling, an extensible classification system aligned with Open Geospatial Consortium (OGC) standards, and detailed caption generation. Additionally, we propose a CLIP-based Image Semantic Enhancement Network (CISEN) to enhance sophisticated ITR capabilities. CISEN comprises dual-path knowledge transfer and progressive cross-modal feature fusion. The former transfers multimodal knowledge from a large, pretrained CLIP-like model, while the latter enhances visual-to-text alignment and fine-grained cross-modal feature integration. Comprehensive statistics on LuojiaHOG demonstrate its richness in sampling diversity, label quantity, and description granularity. Evaluations of LuojiaHOG using various state-of-the-art ITR models–including ALBEF, ALIGN, CLIP, FILIP, Wukong, GeoRSCLIP, and CISEN-employ second- and third-level labels. Adapter-tuning shows that CISEN outperforms others, achieving the highest scores with WMAP@5 rates of 88.47% and 87.28% on third-level ITR tasks, respectively. Moreover, CISEN shows improvements of approximately 1.3% and 0.9% in WMAP@5 over its baseline. When tested on previous RS ITR benchmarks, CISEN achieves performance close to the state-of-the-art methods. Pretraining on LuojiaHOG can further enhance retrieval results. These findings underscore the advancements of CISEN in accurately retrieving relevant information across images and texts. LuojiaHOG and CISEN can serve as foundational resources for future research on RS image–text alignment, supporting a broad spectrum of vision-language applications. The retrieval demo and dataset are available at:https://huggingface.co/spaces/aleo1/LuojiaHOG-demo.