Predicting the effect of pharmacist’s communication with patients: medical communication analysis using facial responses

To provide patient-centred care, psychological knowledge, and skills are necessary for pharmaceutical communication. Acquisition of these communication skills is closely related to patient comprehension. Therefore, to improve pharmacist’s communication skills, pharmacist need to learn the characteristics of their medication instructions, such as posture, facial expressions, eye contact, nodding, and more. For the analysis of medical communication, there is a rating scale, functional analysis, and others. However, these methods may not match the actual emotions due to their analysis skills and the psychological stress of the patients. In this study, we examined the methods to evaluate patient-pharmacist communication using emotion recognition AI software, which recognises emotions from facial expressions. With the cooperation of six simulated patients (SP) and eight pharmacists, we recorded the SP’s facial expressions during medication instruction. The facial expression video was analysed using emotion recognition AI, which can obtain emotion values (anger, contempt, disgust, fear, joy, sadness, surprise, and engagement). We compared the emotion of the extracted peaks with the feedback and calculated the emotion match rate. As a result, 33% of the emotions matched in the peak and feedback. This result indicates that emotion recognition AI cannot capture every feedback emotion. However, in joy, the result was not affected by engagement, and the match rate between peak and feedback was high. In the future, emotion recognition AI will allow us to assess the effects of communication skills of the pharmacists on the psychological state of the patients more objectively and noninvasively.