The contribution of the visual modality to vowel perception in native and non-native speakers

Abstract

This study examines how visual cues impact the intelligibility of foreign-accented speech for native listeners. English CVC words with vowels involving salient (e.g., lip-spreading for /iː/) and non-salient visual cues (neutral lips for /ɪ/) produced by two French speakers and a native English control, were presented to native English listeners who identified the word heard. Tokens were presented in both auditory-only and audiovisual (AV) mode in cafeteria noise at −15 dB SNR. The visual cues analysed were lip spreading, lip rounding, jaw opening and tongue frontness in vowels, as well as lip-rounding in schwa. Visually salient cues improved vowel intelligibility, compared to non-visual cues, but the audiovisual benefit varied across vowel features and speaker groups. The presence of lip-spreading for /iː/ (vs /ɪ/) and jaw-opening for /æ/ (vs /ɪ/) enhanced intelligibility (i.e., larger AV benefit) for both speaker groups. However, compared to the English speaker, lip-rounding in /ɔː/ and /ə/ produced by French speakers (likely accompanied by lip protrusion) had a smaller, or negative AV benefit. These results suggest that the influence of L1 gestures on L2 production may reduce or negatively affect intelligibility. Furthermore, French productions of /ɑː/ exhibited unusually high AV benefits, suggesting an extreme jaw-opening for this vowel in an attempt to distinguish between L2 contrasts (/æ ʌ ɑː/) not present in the L1.