Enhancing spatial perception and contextual understanding for 3D dense captioning

Abstract

3D dense captioning (3D-DC) transcends traditional 2D image captioning by requiring detailed spatial understanding and object localization, aiming to generate high-quality descriptions for objects within 3D environments. Current approaches struggle with accurately describing the spatial location relationships of the objects and suffer from discrepancies between object detection and caption generation. To address these limitations, we introduce a novel one-stage 3D-DC model that integrates a Query-Guided Detector and Task-Specific Context-Aware Captioner to enhance the performance of 3D-DC. The Query-Guided Detector employs an adaptive query mechanism and leverages the Transformer architecture to dynamically adjust attention focus across layers, improving the model’s comprehension of spatial relationships within point clouds. Additionally, the Task-Specific Context-Aware Captioner incorporates task-specific context-aware prompts and a Squeeze-and-Excitation (SE) module to improve contextual understanding and ensure consistency and accuracy between detected objects and their descriptions. A two-stage learning rate update strategy is proposed to optimize the training of the Query-Guided Detector. Extensive experiments on the ScanRefer and Nr3D datasets demonstrate the superiority of our approach, outperforming previous two-stage ‘detect-then-describe’ methods and existing one-stage methods, particularly on the challenging Nr3D dataset.