Multi-scale Local Region Relation Attention in Convolutional Neural Networks for Facial Action Unit Intensity Prediction

Abstract

Facial Action Unit (FAU) intensity can describe the degree of change in the appearance of a specific location on the face and can be used for the analysis of human facial behavior. Due to the subtle changes in FAU, FAU intensity prediction still faces great challenges. Previous works using attention mechanisms for FAU intensity prediction either simply crop the FAU regions or directly use attention mechanisms to obtain local representations of FAUs, but these methods do not capture FAU intensity features at different scales and locations well. In addition, the dependencies between FAUs also contain important information. In this paper, we propose a multi-scale local-region relational attention model based on convolutional neural networks (CNN) for FAU intensity prediction. Specifically, we first reflect the relationship between FAUs by adjusting the luminance values of face images to capture local features with pixel-level relationships. Then, we use the introduced multi-scale local area relational attention model to extract the local attention latent relational features of FAU. Finally, we combine local attention potential relationship features, facial geometry information, and deep global features captured using an autoencoder to achieve robust FAU intensity prediction. The method is evaluated on the public benchmark dataset DISFA, and experimental results show that our method achieves comparable performance to state-of-the-art methods and validates the effectiveness of a multi-scale local-region relational attention model for FAU intensity prediction.