Implementations of the Gesture-based Immersive Interaction and ML-driven Sound Generation for Haptic Companion Doll

The main purpose of virtual reality is to immerse a user in a virtual environment. However, the most common criticism of companion robotics is that they are not able to provide effective tactile feedback. In this research, we will propose an interactive companion doll that has a tail mechanism controlled by the user’s hand gestures as well as generate sound feedback by applying OSC and Wekinator. For the VR content, we designed six emotions for the companion doll to be able to provide the user with different feelings while they are interacting with the companion doll in the virtual and physical worlds.