Exploiting the importance of vowelized sounds in speech comprehension: An application for assisting speechless people

Abstract

This paper proposes a solution that can be used to develop an assistive device to help people that have tracheostomies to make use of verbal communication. Based on the analysis of video images, we propose to develop a system suitable to be used to produce vowel sounds. Exploring the McGurk effect, we expect seeing the mouth movements and listening to the corresponding vowels to be sufficient for improving the understanding of “speechless people”. We conducted three experiments which results show that: 1) just adding vowels' sounds can improve the understandability of the lip reading process; 2) it is possible to detect and to classify vowel-related mouth movements; 3) people can adapt mouth movements to make the system generate the intended vowel, even when it was initially adapted to another person.