Phrasal Synchronization of Gesture With Prosody and Information Structure.

Abstract

This study investigates the synchronization of manual gestures with prosody and information structure using Turkish natural speech data. Prosody has long been linked to gesture as a key driver of gesture-speech synchronization. Gesture has a hierarchical phrasal structure similar to prosody. At the lowest level, gesture has been shown to be synchronized with prosody (e.g., apexes and pitch accents). However, less is known about higher levels. Even less is known about timing relationships with information structure, though this is signaled by prosody and linked to gesture. The present study analyzed phrase synchronization in 3 hr of narrations in Turkish annotated for gesture, prosody, and information structure-topics and foci. The analysis of 515 gesture phrases showed that there was no one-to-one synchronization with intermediate phrases, but their onsets and offsets were synchronized. Moreover, information structural units, topics, and foci were closely synchronized with gesture phrase medial stroke + post-hold combinations (i.e., apical areas). In addition, iconic and metaphoric gestures were more likely to be paired with foci, and deictics with topics. Overall, the results confirm synchronization of gesture and prosody at the phrasal level and provide evidence that gesture shows a direct sensitivity to information structure. These show that speech and gesture production are more connected than assumed in existing production models.