Adaptive Alignment and Time Aggregation Network for Speech-Visual Emotion Recognition

Abstract

Video-based speech-visual emotion recognition plays a crucial role in human-computer interaction applications. However, it faces several challenges, including: 1) the redundancy in the extracted speech-visual features caused by the heterogeneity between speech and visual modalities, and 2) the ineffective modeling of the time-varying characteristics of emotions. To this end, this paper proposes an adaptive alignment and time aggregation network (AataNet). Specifically, AataNet designs a low redundancy speech-visual adaptive alignment (LRSVAA) module to acquire the low-redundant aligned features of speech-visual modalities. Meanwhile, AataNet also designs a computationally efficient time-adaptive aggregation (CETAA) module to model the time-varying characteristics of emotions. Experiments on RAVDESS, BAUM-1 s and eNTERFACE05 datasets also demonstrate that the proposed AataNet achieves better results.