Enhancing multimodal-input object goal navigation by leveraging large language models for inferring room–object relationship knowledge

Abstract

Object-goal navigation is a task in embodied AI where an agent is navigated to a specified object within unfamiliar indoor scenarios. This task is crucial for engineering activities such as training agents in 3D simulated environments and deploying these models in actual mobile robots. Extensive research has been conducted to develop various navigation methods, including end-to-end reinforcement learning and modular map-based approaches. However, fully enabling an agent to perceive and understand the environment, and to navigate towards a target object as efficiently as humans, remains a considerable challenge. In this study, we introduce a data-driven and modular map-based approach, trained on a dataset incorporated with common-sense knowledge of object-to-room relationships extracted from a Large Language Model (LLM), aiming to enhance the efficiency of object-goal navigation. This approach enables the agent to seek the target object in rooms where it is commonly found (e.g., a bed in a bedroom, a couch in a living room), according to LLM-based common-sense knowledge. Additionally, we employ the multi-channel Swin-Unet architecture for multi-task learning, integrating multimodal sensory inputs to effectively extract meaningful features for spatial comprehension and navigation. Results from the Habitat simulator show that our framework surpasses the baseline by an average of 10.6% in the Success-weighted by Path Length (SPL) efficiency metric. Real-world demonstrations confirm that our method can effectively navigate multiple rooms in the object-goal navigation task. For further details and real-world demonstrations, please visit our project webpage (https://sunleyuan.github.io/ObjectNav).