Measuring Affective State: Subject-Dependent and -Independent Prediction Based on Longitudinal Multimodal Sensing

Abstract

Current sensors offering passive and continuous monitoring of behavioral patterns potentially enable real-time affective state monitoring. Previous research on affective state prediction with multimodal sensing in daily life has shown only small-to-moderate effects. One reason for this limited success might be the large variability across individuals. Current research is often of short duration, preventing proper within-individual modeling. With an extensive longitudinal data collection of nine months, this research focuses on individual-level predictions of valence and arousal in daily life. Sixteen PhD candidates from The Netherlands provided data about their affective states (self-reported valence and arousal), physiology (Oura rings) and behavioral patterns (AWARE framework for mobile phone data). Supporting our hypothesis, subject-dependent random forest (RF) models significantly outperformed subject-independent leave-one-subject-out (LOSO) models in predicting self-reported valence and arousal. The subject-dependent models achieved an average Spearman's rho correlation of 0.30 [0.14-0.60] for valence and 0.36 [0.16-0.69] for arousal. In many cases, participants’ a priori indicated informative sources matched with the feature importance. Making use of participants’ self-knowledge might thus help to reduce the amount of data to be collected. For future work, longer-term changes in affective state and combinations of features for estimating real behavioral patterns should be explored.