Hand PointNet-based 3D Hand Pose Estimation in Egocentric RGB-D Images

Abstract

Recently, understanding hand and object manipulation is an active topic in First-Person Vision (FPV) community. In this study, we present an initial study on estimating 3-D hand joints using the state-of-the-art neuronal network. We firstly propose a pre-processing step that is to separate hand regions from clustered background. We deploy the completed pipeline for estimating 3-D hand joints based on HandPointNet (HPN). HPN demonstrates the state-of-the-art hand pose estimation performances with depth data. We deploy a fine-tuning scheme to Hand PointNet (HPN) on the CVAR [1], UCI-EGO [2] datasets for 3D hand pose estimation. In the experimental results, we evaluate the estimated results using the pre-processing step to see the effectiveness of the proposed method. The results show that 3-D joint estimation errors are decreased comparing with the full hand data on different datasets as MSRA, NYU, ICVL. Particularly, we measure the estimation errors of missing, obscured data. The experimental results infer that, it is still existing a big gap between the results of un-occluded and occluded cases. Based on this initial study, we tend to investigate more deeply the techniques addressing the object occlusions or self-occlusions cases that make the current networks hard to localize hidden joints/parts of the hand.