STA-Net: Spatial–temporal alignment network for hybrid EEG-fNIRS decoding

Abstract

Hybrid brain–computer interfaces (BCI) have garnered attention for the capacity to transcend the constraints of single-modality BCI. It is essential to develop innovative fusion methodologies to exploit the high temporal resolution of electroencephalography (EEG) and the high spatial resolution of functional near-infrared spectroscopy (fNIRS). We propose an end-to-end Spatial–Temporal Alignment Network (STA-Net) that achieves precise spatial and temporal alignment between EEG and fNIRS. STA-Net comprises two sub-layers: the fNIRS-guided Spatial Alignment (FGSA) layer and the EEG-guided Temporal Alignment (EGTA) layer. The FGSA layer calculates spatial attention maps from fNRIS to identify sensitive brain regions and spatially aligns EEG with fNIRS through the weighting of EEG channels. The EGTA layer generates temporal attention maps based on the cross-attention mechanism, thereby producing fNIRS signals that are temporally aligned with EEG. This resolves the issue of temporal mismatch caused by the inherent delay of fNIRS. Finally, spatio-temporally aligned EEG-fNIRS signals are fused to classify mental tasks: motor imagery (MI), mental arithmetic (MA), and word generation (WG). STA-Net achieves remarkable performance, with an average accuracy of 69.65% for MI, 85.14% for MA, and 79.03% for WG in subject-specific evaluations, which is superior to state-of-the-art single-modality and multi-modality algorithms. Moreover, STA-Net exhibits less performance degradation in the early stages of tasks compared with the benchmark methods. The spatial–temporal alignment between EEG and fNIRS enhances the performance of hybrid BCI and promotes the decoding of EEG-fNIRS. STA-Net has the potential to establish a new backbone for EEG-fNIRS BCI. The code is available at https://github.com/MutianLiu-SHU/STA-Net.