Learning to estimate 3D interactive two-hand poses with attention perception

Abstract

3D hand pose estimation has attracted increasing research interest due to its broad real-world applications. While encouraging performance has been achieved in single-hand cases, 3D hand-pose estimation of two interactive hands from RGB images still faces two challenging problems: severe intra-hand and inter-hand occlusion and ill-posed projection from 2D hand images to 3D hand joints. To address this, in this paper, we propose a Decoupled Dual-branch Attention Network (DDANet) for 3D interactive two-hand pose estimation. First, we extract multiscale shallow feature maps via a ResNet backbone. Then, we simultaneously learn the context-aware 2D visual and 3D depth features of two interactive hands via two separate attention branches to extensively exploit the two-hand occluded semantic information from RGB images. After that, we define learnable feature vectors to perceive the 3D spatial positions of two-hand joints by interacting them with both 2D visual and 3D depth feature maps. In this way, ill-posed hand-joint positions can be characterized in 3D spaces. Furthermore, we refine the 3D hand-joint spatial positions by capturing the underlying hand-joint connections via GCN learning for 3D two-hand pose estimation. Experimental results on five public datasets show that the proposed DDANet outperforms most state-of-the-art methods with promising generalization.