Exploring the Performance of Facial Expression Recognition Technologies on Deaf Adults and Their Children

Facial and head movements have important linguistic roles in American Sign Language (ASL) and other sign languages. Without being properly trained, both human observers and existing emotion recognition tools will misinterpret ASL linguistic facial expressions. In this study, we capture over 2,000 photographs of 15 participants: five hearing, five Deaf, and five Children of Deaf Adults (CODAs). We then analyze the performance of six commercial facial expression recognition services on these photographs. Key observations include poor face detection rates for Deaf participants, more accurate emotion recognition for Deaf and CODA participants, and frequent misinterpretation of ASL linguistic markers as negative emotions. This suggests a need to include data from ASL users in the training sets for these technologies.