SST-CRAM: spatial-spectral-temporal based convolutional recurrent neural network with lightweight attention mechanism for EEG emotion recognition

Abstract

Through emotion recognition with EEG signals, brain responses can be analyzed to monitor and identify individual emotional states. The success of emotion recognition relies on comprehensive emotion information extracted from EEG signals and the constructed emotion identification model. In this work, we proposed an innovative approach, called spatial-spectral-temporal-based convolutional recurrent neural network (CRNN) with lightweight attention mechanism (SST-CRAM). Firstly, we combined power spectral density (PSD) with differential entropy (DE) features to construct four-dimensional (4D) EEG feature maps and obtain more spatial, spectral, and temporal information. Additional, with a spatial interpolation algorithm, the utilization of the obtained valuable information was enhanced. Next, the constructed 4D EEG feature map was input into the convolutional neural network (CNN) integrated with convolutional block attention module (CBAM) and efficient channel attention module (ECA-Net) for extracting spatial and spectral features. CNN was used to learn spatial and spectral information and CBAM was employed to prioritize global information and obtain detailed and accurate features. ECA-Net was also used to further highlight key brain regions and frequency bands. Finally, a bidirectional long short-term memory (LSTM) network was used to explore the temporal correlation of EEG feature maps for comprehensive feature extraction. To assess the performance of our model, we tested it on the publicly available DEAP dataset. Our model demonstrated excellent performance and achieved high accuracy (98.63% for arousal classification and 98.66% for valence classification). These results indicated that SST-CRAM could fully utilize spatial, spectral, and temporal information to improve the emotion recognition performance.