Interactive Musical Game with a Gesture-Controlled Virtual Puppet

Abstract

In this work, we present an interactive musical game for children, developed using the Unreal Engine 4, with gestural control using the Leap Motion device. To each musical note is assigned a specific gesture to be played with the hand and fingers, so that a virtual puppet can dance according to the music, stimulating the improvement of motor control of children while playing, as well as the learning of basic musical structures of any song. A real-time visual and gestural feedback is generated so that children can compare the gestures they make to those that are determined by the notes of a MIDI file, which are represented in a playful way in the game by a sequence of cards containing drawings of hand gestures. In addition, usability tests were conducted and showed that the learning approach through the execution of gestures, as well as the association of musical concepts to the gamefication, pleased the children, who kept their attention, managed to retain the information learned and found the game fun.