Multimodal coordination and pragmatic modes in conversation

Abstract

Language is intrinsically multimodal. Speakers use gestures, prosody, gaze, and facial expressions as cues that complement and expand the meaning expressed in their words. These varied signals operate in remarkably flexible coordination, constantly adapting to the conversational partners and topics as they change over time. We argue that an ecological approach to multimodal behavior offers a promising account of natural conversation as it takes place both in experimental contexts, and in natural ones outside the lab. After reviewing major historical themes in the study of language and communication, we describe how this ecological perspective situates future work, especially work that seeks to quantify these processes. We describe a quantitative hypothesis that multimodal signals are projected on manifolds of lower dimension that can be described in terms of dynamical systems. We refer to these lower dimensional patterns as “pragmatic modes,” and compare this idea to a number of prior theoretical proposals. We describe how the notion of pragmatic mode frames a quantitative basis to supplement and extend prior research with explicitly quantitative goals. The paper concludes with an outline to link quantitative descriptions of multimodality with more abstract, qualitative theories of the past few decades, and describe how future research might explore pragmatic modes, how they change over the course of conversation, and relate to our understanding of human communication.