Audio-Visual Emotion Recognition System Using Multi-Modal Features

Due to the highly variant face geometry and appearances, facial expression recognition (FER) is still a challenging problem. CNN can characterize 2D signals. Therefore, for emotion recognition in a video, the authors propose a feature selection model in AlexNet architecture to extract and filter facial features automatically. Similarly, for emotion recognition in audio, the authors use a deep LSTM-RNN. Finally, they propose a probabilistic model for the fusion of audio and visual models using facial features and speech of a subject. The model combines all the extracted features and use them to train the linear SVM (support vector machine) classifiers. The proposed model outperforms the other existing models and achieves state-of-the-art performance for audio, visual, and fusion models. The model classifies the seven known facial expressions, namely anger, happy, surprise, fear, disgust, sad, and neutral, on the eNTERFACE’05 dataset with an overall accuracy of 76.61%.