EVA: Key values eclosion with space anchor used in hand pose estimation and shape reconstruction

Abstract

3D hand pose estimation and shape reconstruction from single RGB image face challenges of self-occlusion, object occlusion, and depth ambiguity. Previous methods tried efforts to detect relevant information from images directly. Differently, this paper considers the task as a union of detection and generation. A novel framework called Key Value Eclosion is proposed. It utilizes powerful Diffusion generation strategies to gradually generate and refine occluded joints, vertices, and depth, using visible 2D joint locations as clues. To make the latent codes more comprehensive for hand shape reconstruction, 2D image features are transformed into 3D space using the proposed Space Anchor based feature inverse projection strategy. Integrating the Space Anchor based feature inverse projection into the Key Values Eclosion framework, a complete hand pose estimation and shape reconstruction model called EVA is constructed. The EVA model demonstrates excellent accuracy on both aligned and unaligned metrics using the HO-3D and DexYCB datasets. Especially, the improvement on Mean Error and Trans&Scale metrics are about 30%~50%, compared to state-of-the-art methods.