Facial Expression Recognition With Vision Transformer Using Fused Shifted Windows

Facial expressions contain massive affective information. Previous methods have focused on using diverse models based on CNN or Transformer to handle the facial expression recognition(FER) task. However, most of them treat the FER task as a general image classification task and neglect the impact of regions of interest (ROIs) for the performance of FER. To verify the influence of different ROIs, in this paper, we propose a vision Transformer based on Fused Shifted windows (FSwin), called FSwin Transformer. The semantic information of the face, obtained by ROIs, guides the FSwin Transformer to focus more on the key regions. The fused shifted windows enable the model to perform global semantic interactions, allowing it to concentrate on key regions without losing the topological structural information of the entire face. Additionally, learnable parameters are introduced to learn the feature weights expressed by each ROI, helping the model dynamically adjust the attention distribution. We have conducted controlled experiments to quantitatively verify the impact of ROIs. And the experimental results show that with the increase of the number of ROIs, the accuracy of FER is significantly improved, demonstrating that the key ROIs play an important role in feature extraction. The results on Jaffe, CK+, FER2013, AffectNet have reached 99.8%, 99.0%, 74.8%, and 68.9%, respectively, which all set new state-of-the-art. Extensive cross-dataset experiments also show that the FSwin Transformer has good generalization ability, proving that our proposed model has a beneficial effect on FER tasks.