Primitive audiovisual integration of speech.

Abstract

An unintelligible video recording of a face uttering a sentence and an unintelligible acoustic sinusoid following the frequency variation of a single vocal resonance of the utterance were intelligible when presented together at their veridical synchrony. The intelligibility resulted from audiovisual sensory integration and phonetic perceptual analysis, which depended neither on the separate resolution of linguistic impressions in each modality nor on closed-set reports about a single pair of minimal phonemic contrast features. Likewise, audiovisual integration could not be attributed to Gestalt-derived similarity principles applied unimodally or bimodally. A single sinusoid lacks natural vocal quality, yet was perceptually coherent with the visible form of a natural articulating face. In other tests, integration also survived desynchronization of the two sensory streams, though only if the visible component led the audible component. These measures expose the robust action and temporal dynamic of early non-phonetic sensory integration which can occur in audiovisual speech perception.