Multi-perspective human robot interaction through an augmented video interface supported by deep learning

As the world surpasses a billion cameras [1] and their coverage of the public and private spaces increases, the possibility of using their visual feed to not just observe, but to command robots through their video becomes an ever more interesting prospect. Our work deals with multi-perspective interaction, where a robot autonomously maps image pixels from reachable cameras to positions on its global coordinate space. This enables an operator to send the robot to specific positions in a camera with no manual calibration. Furthermore, robot information, such as planned paths, can be used to augment all affected camera images with an overlayed projection of their visual information. The robustness of this approach has been validated in both simulated and real world experiments.