Introducing motion-capturing technology into the music practice room as a feedback tool for working towards the precision ofrubato

Abstract

Motion-capturing technology has been traditionally used in the field of sports for the analysis of athletes’ body movements. The application of this multimodal tool to the field of musical pedagogy, however, has yet to be widely explored. Historically, music teachers have been using abstract language such asthick lines, for example, to describe melodic phrases. Thesethick linescan function as mental signposts or cues to help a performer navigate through a particular piece especially during performances. Concepts such asrubato– the stretching of time – for instance, are more challenging to describe in terms of concrete lines or shapes, however. Because the push and pull ofrubatois so subtle, it can sometimes be challenging to pinpointrubatoand maximize its effectiveness. Thus, in addition to listening to the coach’s verbal explanations, it can be helpful to see the teacher’s gestures displayed simultaneously alongside their students’ gestures during music lessons. TheLeapmotionsoftware provides visual feedback in real time and can be played back in slow motion. This device functions much like a mirror, as the performers’ gestures reflect onto the screen in real time. At the same time, their teacher’s gestures can also be juxtaposed onto the screen as a reference. Details of the speed and the precision of therubatotiming can be seen on the screen as well. More importantly,Leapmotioncan be a useful source of feedback in the practice room where the teacher is not present; students can record precise gestures during their lessons and revisit what they had learned when they are alone. In this sense, students would not feel lost in the practice room during the week, and they could also hone their music analysis skills through the examination of their body movements. This study aims to catalyse the learning process and to revolutionize the traditional methods of daily music practice.