Enhancing collaborative signing songwriting experience of the d/Deaf individuals

Songwriting can be an important means of developing the personal and social skills of d/Deaf individuals, but there is a lack of research on understanding and supporting their songwriting. We aimed to understand the d/Deaf people's songwriting experience for the song signing genre, which visually represents music with sign language and body movement. Through two workshops in which mixed-hearing individuals collaborated in songwriting activities, we identified the potentials and challenges of the songwriting experience and developed a music-sensory substitution system that multimodally presents music in sound as well as visual, and vibrotactile feedback. The proposed system enables mixed-hearing partners to have better collaborative interaction and signing songwriting experience. Consequently, we found that the process of signing songwriting is valued by d/Deaf individuals as a means of musical self-expression and social connecting, and our system has increased their musical engagement while encouraging them to express themselves more through music and sign language.