Multimodal marking of information structure: gesture-prosody alignment across languages

Abstract

In this paper, we first review the existing evidence of gesture-prosody alignment in information structure marking, focusing on specific gestural patterns that were observed to co-occur with various information structure constructions. Then we complement the evidence with the results of a corpus-based study of gesture-speech alignment in Czech. Analyzing a sample of 80 minutes of personal narratives by 16 speakers collected from a Czech multimodal corpus, we observed that by far the most frequent information structure units accompanied by gestures were foci. In line with previous research, we observed that pitch and intensity peaks lag behind the gesture stroke onset (on average by 300 ms). We also provide new evidence for a systematic variation in the duration of the temporal shift related to the marking of discourse contrast. represents the first survey of multimodal marking of information structure in Czech. As such, it provides rather general first-look observations, as we employed relatively coarse-grained measures that did not allow for a closer ex-amination of the gesture-prosody-discourse entanglement. Further investigation of our corpus data has to consider possible alignment not only to F0 peaks but also phrase boundaries or low-pitch tones. However, the approach taken in this is a good starting point. Also, as pointed out by Türk (2020), it is not necessarily only the apex of a gesture stroke that should be mapped to prosodic markers. For instance, we should focus on whether — and if so, in what contexts — post-stroke holds follow strokes that are out of phase with pitch contour. for her assistance with the annotation and Martin Sedláček for proofreading the manuscript.