Rhythm as an integration principle for modeling speech-action intersemiosis in classroom interaction: a social semiotic perspective

Abstract

A continuing challenge for scholars working with multimodal educational research is to devise theoretical and methodological tools that can effectively navigate the complexity and emergent meaning when different semiotic resources interact. This paper demonstrates how rhythm, as an integration principle, coordinates the interaction of speech and embodied action in classroom settings at multi-scalar temporalities. Transcription designs are also devised to capture and visualize the patterns of multimodal rhythmic interaction. Drawing on a social semiotic theorization of rhythm, the paper conducts nuanced multimodal analyses of video data documenting teacher-student embodied interaction. The paper first reports four types of multimodal rhythmic patterns in classroom interaction, showcasing how rhythms coordinate across participants and semiotic resources. It then demonstrates how the tempo of the speech rhythmically structures the embodied actions at different time scales, resulting in multimodal synchronies that are semantically motivated. Finally, the paper reveals that the multiple actions in a pedagogic practice, while themselves rhythmical, may not always be rhythmically integrated with speech. The paper contributes to existing studies of speech-action interplay by developing theoretical and methodological tools to capture and visualize their interactions. Observations developed in this paper can also potentially inform pedagogic practices that involve the co-deployment of speech and embodied action.