[Regular Paper] KnowPain: Automated System for Detecting Pain in Neonates from Videos

Premature neonates are subjected to clinically required but painful procedures throughout their hospitalization. Since neonates are non-verbal, pain scoring tools are used to measure their pain responses. Although a number of pain instruments have been developed to assist health professionals, these tools are subjective and may underestimate the pain response of neonates. This could lead to the pain being misread resulting in mis-diagnosis and under/over treatment. In this paper, a deep learning based approach is used to detect pain in videos of premature neonates during painful clinical procedures. A Conditional Generative Adversarial Network (CGAN) is used to continuously learn the representation and classify painful facial expressions in neonates from real and synthetic data. A Long Short-Term Memory (LSTM) is used for modeling the temporal changes in facial expression to further improve the classification. Furthermore, the proposed approach is able to implicitly learn the intensity of pain as a probability score directly from the facial expressions without any manual annotation. Experimental results show that this approach achieves an accuracy of 95.34% on the iCOPE Classification Of Pain Expressions (video) dataset, 88.27% on the Loma Linda Infant Pain Expressions (video) dataset and 94.12% on the Infant Classification Of Pain Expressions (static images) dataset outperforming state-of-the-art approaches.