Sonification of 3D body movement using parameter mapping technique

Abstract

Most of instructions by trainers or therapists in body movements, such as walking, turning, rising arms or legs are mostly done through voice instructions or touches. This does not have much problem to normal people as they can see it at the same time. To follow these instructions without seeing it can cause confusion in terms of actions and directions. Unfortunately, there is no other option for people who are blind. Thus, this research will try to replace those voice and touches form of instructions into non-speech sound instructions. The method involves transforming 3-dimensional data of body movements (kinematics) into sounds. It is hoped by only listening to the sounds, a person should be able to follow the exact movements of the body of another person or instructor without any voice commands or instructions. The novel contribution of this research is to produce an effective and efficient sonification technique (converting the data into sound) to represent the actions and directions of the body movements in 3-dimensional space. The conversion approach to be used is Parameter Mapping, where the movement properties will be mapped to sound properties. The parameter mapping will involve at least 3 transformation processes - data, acoustics parameters and sound representations. The effectiveness and efficiency of this approach is depending on all of these transformation processes. This research will use Kinect (a device used to play game with Microsoft Xbox game console) as the live 3D movements input data stream. This device is intentionally used in this research, as it is readily available and cheap. Thus, the potential applications or products from this research later on can also use this device.