Spatio-Temporal Action Order Representation for Mobile Manipulation Planning*

Social robots are used to perform mobile manipulation tasks, such as tidying up and carrying, based on instructions provided by humans. A mobile manipulation planner, which is used to exploit the robot’s functions, requires a better understanding of the feasible actions in real space based on the robot’s subsystem configuration and the object placement in the environment. This study aims to realize a mobile manipulation planner considering the world state, which consists of the robot state (subsystem configuration and their state) required to exploit the robot’s functions. In this paper, this study proposes a novel environmental representation called a world state-dependent action graph (WDAG). The WDAG represents the spatial and temporal order of feasible actions based on the world state by adopting the knowledge representation with scene graphs and a recursive multilayered graph structure. The study also proposes a mobile manipulation planning method using the WDAG. The planner enables the derivation of many effective action sequences to accomplish the given tasks based on an exhaustive understanding of the spatial and temporal connections of actions. The effectiveness of the proposed method is evaluated through practical machine experiments performed. The experimental result demonstrates that the proposed method facilitates the effective utilization of the robot’s functions.