Multi-Modal Fused-Attention Network for Depression Level Recognition Based on Enhanced Audiovisual Cues

In recent years, substantial research has focused on automated systems for assessing depression levels using different types of data, such as audio and visual inputs. However, signals recorded from individuals with depression can be influenced by external factors, such as the recording equipment and environment, making it essential to create a system that is resilient to these interferences to maintain accuracy. This study introduces a fused-attention model for evaluating depression severity using enhanced multi-modal data inputs. Applying several pre-trained advanced models, this article incorporates audiovisual sequences with augmentation. The framework includes two novel components, which we term as the FIE and VIE blocks, for extracting detailed facial and vocal features. The FIE block utilizes ResNet-18 to enhance the feature representation of video frames and integrates two types of attention mechanisms to capture spatial-temporal patterns. Meanwhile, the VIE block processes the Mel spectrogram of the audio signal, followed by an optimized Swin transformer block to extract auditory features. The model demonstrates strong performance, accurately identifying depression severity in 3-second audiovisual sequences with an 81.4% accuracy rate on the AVEC2014 dataset, and achieves a Kappa score of 0.731 and an MF1 index of 0.798. Furthermore, it shows high resilience to noise, underscoring its ability to mitigate the effects of recording equipment and environmental conditions in depression level estimation.