Integrated analysis of speech and images as a probabilistic decoding process

Abstract

Speech understanding and vision are the two most important modalities in human-human communication. However, the emulation of these by a computer faces fundamental difficulties due to noisy data, vague meanings, previously unseen objects or unheard words, occlusions, spontaneous speech effects, and context dependence. Thus, the interpretation processes on both channels are highly error-prone. This paper presents a new perspective on the problem of relating speech and image interpretations as a probabilistic decoding process. It is shown that such an integration scheme is robust regarding partial or erroneous interpretations. Furthermore, it is shown that implicit error correction strategies can be formulated in this probabilistic framework that lead to improved scene interpretation.