Individual-Aware Attention Modulation for Unseen Speaker Emotion Recognition

Abstract

In practical human-computer interaction (HCI) applications, robust speech emotion recognition (SER) for unseen speakers is crucial. Prior research has primarily focused on extracting common representations to enhance the generalization of cross-individual SER. However, most methods ignore the positive effects of individual characteristics. Actually, each speaker can be regarded as an independent individual domain. Personalized SER can be improved if the emotional expressions of individual speech characteristics are effectively utilized. To address the challenges in recognizing emotions for unseen speakers, this paper proposes a novel individual-aware attention modulation (IAM) model. Specifically, the IAM uses meta-learning techniques to extract modulation parameters for obtaining individual-related emotion expressions from individual characteristics. The base model is then modulated to facilitate the transfer of the common emotion representation space to an individual-specific emotion representation space. This transformation is achieved by applying attention modulation within the transformer-based model developed in this paper. In addition, we employ a meta-learning-based method to optimize model parameters, enhancing the adaptability of the model to unseen speakers, and a control factor is introduced to regulate the degree of individual modulation, thus enhancing the robustness of the modulation process. Experimental results demonstrate that the proposed model achieves significantly improved cross-individual SER performance.