Proactive Robot Assistants for Freeform Collaborative Tasks Through Multimodal Recognition of Generic Subtasks

Successful human-robot collaboration depends on a shared understanding of task state and current goals. In nonlinear or freeform tasks without an explicit task model, robot partners are unable to provide assistance without the ability to translate perception into meaningful task knowledge. In this paper, we explore the utility of multimodal recurrent neural networks (RNNs) with long short-term memory (LSTM) units for real-time subtask recognition in order to provide context-aware assistance during generic assembly tasks. We train RNNs to recognize specific subtasks in individual modalities, then combine the high-level representations of these networks through a nonlinear connection layer to create a multimodal subtask recognition system. We report results from implementing the system on a robot that uses the subtask recognition system to provide predictive assistance to a human partner during a laboratory experiment involving a human-robot team completing an assembly task. Generalizability of the system is evaluated through training and testing on separate tasks with some similar subtasks. Our results demonstrate the value of such a system in providing assistance to human partners during a freeform assembly scenario and increasing humans' perception of the robot's agency and usefulness.