A Cloud-Based Robust Semaphore Mirroring System for Social Robots

We present a cloud-based human-robot interaction system that automatically controls a humanoid robot to mirror a human demonstrator performing flag semaphores. We use a cloud-based framework called Human Augmented Robotic Intelligence (HARI) to perform gesture recognition of the human demonstrator and gesture control of a local humanoid robot, named Pepper. To ensure that the system is real-time, we design a system to maximize cloud computation contribution to the deep-neural-network-based gesture recognition system, OpenPose, and to minimize communication costs between the cloud and the robot. A hybrid control system is used to hide latency caused by either routing or physical distances. We conducted real-time semaphore mirroring experiments in which both the robots and the demonstrator were located in Tokyo, Japan, whereas the cloud server was deployed in the United States. The total latency was 400ms for the video streaming to the cloud and 108ms for the robot commanding from the cloud. Further, we measured the reliability of our gesture-based semaphore recognition system with two human subjects, and were able to achieve 90% and 76.7% recognition accuracy, respectively, for the two subjects with open-loop when the subjects were not allowed to see the recognition results. We could achieve 100% recognition accuracy when both subjects were allowed to adapt to the recognition system under a closed-loop setting. Lastly, we showed that we can support two humanoid robots with a single server at the same time. With this real-time cloud-based HRI system, we illustrate that we can deploy gesture-based human-robot globally and at scale.