Upper body pose estimation from stereo and hand-face tracking

Abstract

In applications such as immersive telepresence we want to extract high quality 3D models of collaborators in real time from multiview image sequences. One way to improve the quality of stereo or visual hull based models is to estimate the kinematic pose of the user first and then constrain 3D reconstruction accordingly. To serve as a preprocessing step such pose extraction must be very fast, precluding the usual generate and test techniques. We examine a method based on psychophysical evidence that known relative hand position can be used to directly compute the pose of the arm. First we explore a number of possible models for this relationship using motion capture data. We then examine how reconstruction of face and hand position as well as a patch on the torso, allow us to exploit these simple direct calculations to estimate the pose of a user in a desktop collaboration environment.