A human-robot interaction system for automated chemical experiments based on vision and natural language processing semantics

Abstract

Using collaborative robots to replace researchers in performing repetitive and hazardous chemical experiments can effectively enhance experimental efficiency. However, this technology still faces several challenges, including understanding researchers' natural language instructions, autonomously generating action sequences, and more. Therefore, we developed a general control framework for robots in automated chemical experiments based on visual and natural language semantic information. Firstly, starting with the recognition of keywords within Chinese language instructions, we established a domain dictionary for chemical experiment operations and proposed an instruction understanding model based on the bidirectional long-short-term memory and conditional random field(BiLSTM-CRF), enhancing the robot's cognitive ability towards user instructions. Then, a rule matching method for chemical experimental information and a multimodal information feature matching mechanism were established for command content verification and the automatic generation of multiple types of structured language. At the same time, a robot feedback mechanism was added, enabling human-computer interaction and establishing closed-loop control of the system. Finally, propose a robot action sequence generation mechanism based on hierarchical finite state machines(HFSM), transforming structured language into operational strategies for chemical experiments required by the robot. Experimental results show that on the instruction task comprehension dataset created in this paper, the proposed method improves the F1 score by up to 4.44% in the instruction keyword extraction task compared to other models. In addition, compared to traditional manual teaching control, this method significantly reduces time costs. This verifies that the method effectively enhances the robot's ability to comprehend Chinese instructions and generates reliable executable action sequences.