Influence of visual analogue of speech envelope, formants, and word onsets on word recognition is not pronounced

Abstract

In noisy environments, filtering out the relevant speech signal from the background noise is a major challenge. Visual cues, such as lip movements, can improve speech understanding. This suggests that lip movements carry information about speech features (e.g. speech envelope, formants, word onsets) that can be used to aid speech understanding. Moreover, the isolated visual or tactile presentation of the speech envelope can also aid word recognition. However, the evidence in this area is rather mixed, and formants and word onsets have not been studied in this context.

This online study investigates the effect of different visually presented speech features (speech envelope, formants, word onsets) during a two-talker audio on word recognition. The speech features were presented as a circle whose size was modulated over time based on the dynamics of three speech features. The circle was either modulated according to the speech features of the target speaker, the distractor speaker or an unrelated control sentence. After each sentence, the participants` word recognition was tested by writing down what they heard. We show that word recognition is not enhanced for any of the visual features relative to the visual control condition.