Emotion knowledge-based fine-grained facial expression recognition

Abstract

Existing facial expression recognition (FER) techniques rely primarily on seven coarse-grained emotions as emotional labels, which are insufficient to cover the subtle changes in human emotions in the real world. We use 135 fine-grained emotions as emotional benchmarks to address the problem of highly semantically similar fine-grained emotion recognition. In this work, we propose a robust emotion knowledge-based fine-grained (EK-FG) emotion recognition network that captures inter-class relationships and discriminative representations of fine-grained emotions through two prior-based losses: coarse-grained hierarchical loss and fine-grained semantic loss. Specifically, the coarse-grained hierarchical loss obtains a structured semantic representation of fine-grained emotions based on prior knowledge, and captures inter-class relationships through effective category-level push-pull to obtain discriminative representations. The fine-grained semantic loss provides more accurate measurement information for semantic features based on prior knowledge, and enhances the model’s discriminative ability for subtle facial expression differences through regression constraints. Extensive experimental results on the Emo135 dataset demonstrate that EK-FG can effectively overcome the class ambiguity of fine-grained emotion.