Multiscale Transformer Hierarchically Embedded CNN Hybrid Network for Visible-Infrared Person Reidentification

Abstract

Visible-infrared person reidentification (VI-ReID) is considered a pivotal technology for intelligent security surveillance systems for the Internet of Things (IoT). For the VI-ReID task, one key challenge is extracting and fusing robust global and local pedestrian information to mitigate the intermodality discrepancy. Despite the significant success achieved by convolutional neural network (CNN)-based methods, the extraction of global pedestrian information is limited by their inherent properties, namely, local receptive fields and downsampling processes, making cross-modality information fusion difficult. While existing pure Transformer-based methods excel at capturing global pedestrian information, uniform-sized queries, keys, and values are employed by their core self-attention mechanism. This results in the acquisition of uniform-scale information only, thereby limiting the learning of multiscale information and preventing the full extraction of local pedestrian information. To address the aforementioned issues, a multiscale Transformer hierarchically embedded CNN hybrid network (MTECN) is proposed by us. MTECN enables the simultaneous extraction of pedestrian local and global information at different scales to mitigate the adverse impact on recognition caused by the discrepancy in features extracted across different modalities. Moreover, the effects of inherent factors, including camera viewpoint and illumination variations, are alleviated by incorporating a spatial consistency (SC) loss, which guides the network in exploring and discriminating the spatial structures of pedestrians across different modalities, consequently aligning the underlying spatial semantic information. Furthermore, in the low-light VI-ReID task, information insufficiency is encountered by the LLCM dataset due to low-light conditions. Consequently, a low-light enhancement (LLE) module is employed to restore the obscured detail information in low-light images, thereby further enhancing MTECN’s robust feature learning in complex backgrounds. To the best of our knowledge, this is the first work to use Transformer hierarchically embedded CNN networks for VI-ReID research, and the first to use LLE techniques for low-light VI-ReID task. Extensive experiments on the SYSU-MM01, RegDB, and LLCM datasets show that the proposed MTECN method excels over several state-of-the-art methods.