Exploring the impact of computer-mediated emotional interactions on human facial and physiological responses

Abstract

Remote communication has become pervasive, yet its impact on human emotions, empathy, and physiological responses remains unclear. This study addresses this gap by investigating how emotional video-mediated conversations differ from face-to-face interactions, focusing on behavioral and physiological responses. We create a multimodal dataset of Electrodermal Activity (EDA) signals, Photoplethysmography (PPG) signals, and facial videos from two people conversing about emotional topics in face-to-face and remote video-mediated conditions. We use a series of repeated measures ANOVA with aligned rank transform to compare face-to-face to remote conversation in terms of heart rate activity, electrodermal activity, and facial action units. We also explore how subjective empathy between people varies in these two conditions. Our findings reveal significant differences in physiological responses between face-to-face and remote conversation and variations in perceived empathy based on interaction setting, highlighting the nuanced influence of communication channels. We also show that we can recognize emotions more accurately when we pre-train a random forest classifier with one condition’s data (an increase of 20% to 45% for various modalities). Finally, we discuss the research findings and limitations and offer insights for optimizing human–computer interaction and understanding human emotional responses in an increasingly tech-mediated world.