Deep Learning based Affective Sensing with Remote Photoplethysmography

Abstract

Recent studies show that heart rate variability (HRV) is an important physiological characteristic that reflects physiological and affective states of a person. Advancements in the field of remote camera-based photoplethysmography has made possible measurement of cardiac signals using just the raw face videos. Most of existing studies of camera-based cardiovascular monitoring focus on just heart rate (HR) estimation, leaving more interesting case of remote HRV estimation out of scope. However, knowing only the average HR is not enough for affective sensing applications, and measurement of HRV is beneficial. We propose a new framework, which uses deep spatiotemporal networks for contactless HRV measurements from raw facial videos. The proposed framework employs data augmentation technique. It was evaluated on two multimodal databases that consists face videos with synchronized physiological signals. Experiments demonstrate the advantage of our deep learning based approach for HRV estimation. We also achieved promising results for inclusion remote HRV estimation in affective sensing applications.