Automatic Depression Recognition With an Ensemble of Multimodal Spatio-Temporal Routing Features

Abstract

Depression, driven by growing societal pressures, significantly disrupts individuals’ physical and mental health. Automatic Depression Recognition (ADR) via facial videos has gained attention to enhance diagnostic accuracy and efficiency. However, extant methods often segment videos, losing long-term behavioral cues and introducing noise, while also exhibiting performance drops across diverse cultural and racial datasets. This study proposes a multimodal ADR approach encompassing three key components: (1) Long-term Depression Behavior Module (LDBM) employing a Transformer to capture extended depression cues, (2) Noisy Information Elimination (NIE) strategy leveraging LDBM attention scores to reduce noise and boost diagnostic precision, and (3) Multimodal Spatio-temporal Routing Feature Ensemble (MSRE) that fuses texture, Facial Action Primitives (FAPs), and Remote Photoplethysmography (rPPG) data for improved cross-dataset generalizability. Experiments on AVEC 2013, AVEC 2014, and a newly constructed CMDep dataset of 123 clinically diagnosed participants validate our method, achieving MAE/RMSE scores of 5.38/6.74, 5.09/6.83, and 5.59/8.03, respectively. The CMDep dataset includes facial expression and voice signals, with labels derived from BDI-II scores. Additionally, our method has been integrated into a user-friendly mobile application, providing a tool for real-time self-assessment of depression. This integration broadens the scope of depression detection, making it accessible to diverse populations worldwide.