Can you read lips with a masked face?

Abstract

We have been working on lip-reading that estimates the content of utterances using only visual information. When most people started wearing masks due to the coronavirus pandemic, several people asked us if using machine-based lip-reading even when wearing a mask was possible. Taking this as a research question, we worked on word-level lip-reading for the world’s first masked face image. The utterance scene dataset when wearing a mask is not open to the public, so we developed our dataset, face detection for masked face images, facial landmarks detection, and lip-reading based on deep learning. We collected speech scenes of 20 people for 15 Japanese words and obtained a recognition accuracy of 88.3% as a result of the recognition experiment. This paper reports that lip-reading is possible for masked face images.