Improving Medical Students’ Awareness of Their Non-Verbal Communication through Automated Non-Verbal Behavior Feedback

The nonverbal communication of clinicians has an impact on patients’ satisfaction and health outcomes. Yet medical students are not receiving enough training on the appropriate nonverbal behaviors in clinical consultations. Computer vision techniques have been used for detecting different kinds of nonverbal behaviors, and they can be incorporated in educational systems that help medical students develop communication skills. We describe EQClinic, a system that combines a tele-health platform with automated nonverbal behavior recognition. The system aims to help medical students improve their communication skills through a combination of human and automatically generated feedback. EQClinic provides fully automated calendaring and video-conferencing features for doctors or medical students to interview patients. We describe a pilot (18 dyadic interactions) in which standardized patients (i.e. someone acting as a real patient), were interviewed by medical students and provided assessments and comments about their performance. After the interview, computer vision and audio processing algorithms were used to recognize students’ nonverbal behaviors known to influence the quality of a medical consultation: including turn taking, speaking ratio, sound volume, sound pitch, smiling, frowning, head leaning, head tilting, nodding, shaking, face-touch gestures and overall body movements. The results showed that students’ awareness of nonverbal communication was enhanced by the feedback information, which was both provided by the standardized patients and generated by the machines.