Achieving adaptive tasks from human instructions for robots using large language models and behavior trees

Abstract

Combining Large Language Models (LLMs) with Behavior Trees (BTs) provides an alternative to achieve robot adaptive tasks from human instructions. BTs that contain goal conditions are generated by LLMs based on user instructions and then expanded by BT planners to accomplish tasks and handle disturbances. However, current LLMs struggle to handle unclear human instructions and have a relatively weak understanding of spatial geometry between objects, which results in suboptimal BT planning. To address these problems, this paper proposes a two-stage framework. In the first stage, a Feedback module is designed to handle unclear user instructions and guide the LLM to communicate with users, thus making the goal conditions of BTs complete. In the second stage, a BT Adaptive Update algorithm is proposed to optimize the execution order of the goal conditions, thereby improving the task efficiency of BT planner for multi-goal tasks. Experimental results from simulations and the real world indicate that our method enables the robot to generate complete goal conditions from user instructions and accomplish multi-goal tasks efficiently.