Learning a Parallel Network for Emotion Recognition Based on Small Training Data

Speech emotion recognition (SER) classifies speech into emotion categories such as “happy”, “sad”, and “angry”. Speech emotion recognition has attracted more and more attention in recent years as a challenging pattern recognition task, but its performance is limited by the amount of training data. In this paper, we propose a parallel network consisting of a CNN and a Transformer that receives two types of inputs. The Convolutional Neural Network (CNN) accurately recognizes emotions from the speech data using a mel-spectrogram feature. The transformer uses Multi-Attention from Mel-Frequency Cepstrum Coefficient (MFCC) to realize the extraction of emotional semantic information in a sequence. Experiments are carried out on the Ryerson Audio-Visual Database of Emotion Speech and Song (RAVDESS) dataset. The results demonstrate the effectiveness of the proposed method and show significant improvement over previous results with fewer data and less training time without data augmentation.