Images, normal maps and point clouds fusion decoder for 6D pose estimation

Abstract

6D pose estimation plays a crucial role in enabling intelligent robots to interact with their environment by understanding 3D scene information. This task is challenging due to factors such as texture-less objects, illumination variations, and scene occlusions. In this work, we present a novel approach that integrates feature fusion from multiple data modalities—specifically, RGB images, normal maps, and point clouds—to enhance the accuracy of 6D pose estimation. Unlike previous methods that rely solely on RGB-D data or focus on either shallow or deep feature fusion, the proposed method uniquely incorporates both shallow and deep feature fusion across heterogeneous modalities, compensating for the information often lost in point clouds. Specifically, the proposed method includes an adaptive feature fusion module designed to improve the communication and fusion of shallow features between RGB images and normal maps. Additionally, a multi-modal fusion decoder is implemented to facilitate cross-modal feature fusion between image and point cloud data. Experimental results demonstrate that the proposed method achieves state-of-the-art performance, with 6D pose estimation accuracy reaching 97.7% on the Linemod dataset, 71.5% on the Occlusion Linemod dataset, and 95.8% on the YCB-Video dataset. These results underline the robustness and effectiveness of the proposed approach in complex environments.