Towards Robust 3D Skeleton Tracking Using Data Fusion from Multiple Depth Sensors

Real-time full-body tracking in VR is important for providing realistic experiences, especially for applications such as training, education, and social VR. The Microsoft Kinect v2 sensor can provide skeleton data for a user in real-time, however, due to occlusion issues and front/back ambiguity errors, one Kinect is not always reliable enough for the correct capture of 360-degree movements. In this paper, we present work to provide robust, real-time tracking using multiple Kinect v2 cameras. An adaptive data fusion method is described that constructs a high-quality 3D skeleton which can be used to drive a VR avatar regardless of the user's orientation. We compare three different approaches to fusing the data from the three Kinects, and compare against ground truth using an OptiTrack system. A static pose and a dynamic movement were captured to compare errors of each joint using the three fusion algorithms. Our results show that an adaptive weighting adjustment fusion method for combining skeleton data from the three Kinects according to the current facing direction performed best in terms of joint error.