Emotion Recognition by Learning the Manifold of Fused Multiscale Information of EEG Signals

Abstract

Recent research has consistently indicated that the fusion of electroencephalography (EEG) features from multiple modalities can integrate cognitive state expressions across diverse dimensions, resulting in a substantial increase in emotion recognition accuracy. However, redundant information within the fused multimodal features could lead to the curse of dimensionality and overfitting of the learning model. In this work, we propose a multiscale EEG feature fusion and representation strategy for EEG emotion recognition named manifold of multiscale information fusion (MMIF), in which the optimal manifold of the multiscale fusion of local and global brain activation patterns can be automatically learned to realize an efficient representation of emotional EEG signals. To evaluate the performance, in this work, both off- and online EEG emotion recognition experiments were conducted, and the experimental results consistently verified the effectiveness and feasibility of the MMIF applied in real-time emotion decoding systems. Furthermore, the analytical experiments confirmed the discriminative capabilities and cognitive interpretability of the MMIF. In summary, the proposed MMIF model may provide an efficient avenue for exploring representations and enhancing the discrimination of multimodal fusion features, which may also provide a promising solution for designing online affective braincomputer interaction systems.