ISPRS Journal of Photogrammetry and Remote Sensing最新文献

On the basis of the transformer architecture and the pretext task of “next-token prediction”, multimodal large language models (MLLMs) are revolutionizing the paradigm in the field of remote sensing image understanding. However, the tokenizer, as one of the fundamental components of MLLMs, has long been overlooked or even misunderstood in visual tasks. A key factor contributing to the great comprehension power of large language models is that natural language tokenizers utilize meaningful words or subwords as the basic elements of language. In contrast, mainstream visual tokenizers, represented by patch-based methods such as Patch Embed, rely on meaningless rectangular patches as basic elements of vision. Analogous to words or subwords in language, we define semantically independent regions (SIRs) for vision and then propose two properties that an ideal visual tokenizer should possess: (1) homogeneity, where SIRs serve as the basic elements of vision, and (2) adaptivity, which allows for a flexible number of tokens to accommodate images of any size and tasks of any granularity. On this basis, we design a simple HOmogeneous visual tOKenizer: HOOK. HOOK consists of two modules: an object perception module (OPM) and an object vectorization module (OVM). To achieve homogeneity, the OPM splits the image into 4 × 4 pixel seeds and then uses a self-attention mechanism to identify SIRs. The OVM employs cross-attention to merge seeds within the same SIR. To achieve adaptability, the OVM predefines a variable number of learnable vectors as cross-attention queries, allowing for the adjustment of the token quantity. We conducted experiments on the NWPU-RESISC45, WHU-RS19, and NaSC-TG2 classification datasets for sparse tasks and the GID5 and DGLCC segmentation datasets for dense tasks. The results show that the visual tokens obtained by HOOK correspond to individual objects, thereby verifying their homogeneity. Compared with randomly initialized or pretrained Patch Embed, which required more than one hundred tokens per image, HOOK required only 6 and 8 tokens for sparse and dense tasks, respectively, resulting in performance improvements of 2% to 10% and efficiency improvements of 1.5 to 2.8 times. The homogeneity and adaptability of the proposed approach provide new perspectives for the study of visual tokenizers. Guided by these principles, the developed HOOK has the potential to replace traditional Patch Embed. The code is available at https://github.com/GeoX-Lab/Hook.

Establishing accurate correspondences between aerial and ground images is facing immense challenges because of the drastic viewpoint, illumination, and scale variations resulting from significant differences in viewing angles, shoot timing, and imaging mechanisms. To cope with these issues, we propose an effective aerial-to-ground feature matching method, named Viewpoint-invariant Deformable Feature Transformation (VDFT), which aims to comprehensively enhance the discrimination of local features by utilizing deformable convolutional network (DCN) and seed attention mechanism. Specifically, the proposed VDFT is constructed consisting of three pivotal modules: (1) a learnable deformable feature network is established by using DCN and Depthwise Separable Convolution (DSC) to obtain dynamic receptive fields, addressing local geometric deformations caused by viewpoint variation; (2) an improved joint detection and description strategy is presented through concurrently sharing the multi-level deformable feature representation to enhance the localization accuracy and representation capabilities of feature points; and (3) a seed attention matching module is built by introducing self- and cross- seed attention mechanisms to improve the performance and efficiency for aerial-to-ground feature matching. Finally, we conduct thorough experiments to verify the matching performance of our VDFT on five challenging aerial-to-ground datasets. Extensive experimental evaluations prove that our VDFT is more resistant to perspective distortion and drastic variations in viewpoint, illumination, and scale. It exhibits satisfactory matching performance and outperforms the current state-of-the-art (SOTA) methods in terms of robustness and accuracy.