An efficient approach for the elevator button manipulation using the visual-based self-driving mobile manipulator

Purpose Because mobile manipulators are unable to climb stairs, the elevator operation is a crucial capacity to help those kinds of robot systems work in modern multifloor buildings. Here, the elevator button manipulation is considered as an efficient approach to fulfill that requirement. Previously, some studies presented elevator button recognition algorithms while some others designed schemes for the button manipulation work. However, the mobile robot, the manipulator and the camera in their robot systems are asynchronous. Besides, the time-consuming calibration for the camera is inevitable, especially in changeable environments. This paper aims to present an alternative method for the elevator button manipulation to overcome mentioned shortcomings. Design/methodology/approach In this paper, the elevator button manipulation is conducted by using the visual-based self-driving mobile manipulator in which the autonomous mobile robot, the manipulator and the camera cooperate more efficiently. Namely, the mobile robot does not need to be located exactly in front of the elevator panel as the manipulator has the ability to adjust the initial frame of the camera based on the system kinematic synchronization. In addition, the proposed method does not require the real world coordinates of elevator buttons, but uniquely using their pixel positions. By doing this, not only is the projection from two-dimensional pixel coordinates to three-dimensional (3D) real world coordinates unnecessary, but also the calibration of the camera is not required. Findings The proposed method is experimentally verified by using a visual-based self-driving mobile manipulator. This robotic system is the integration of an autonomous mobile robot, a manipulator and a camera mounted on the end-effector of the manipulator. Research limitations/implications Because the surface of the elevator button panel is usually mirror-like, the elevator button detection is easily affected by the glare and the brightness of the environmental light condition. Practical implications This robot system can be used for the goods delivery or the patrol in modern multifloor buildings. Originality/value This paper includes three new features: simultaneously detecting and manipulating elevator buttons without the projection from pixel coordinates to 3D real world coordinates, a kinematic synchronization to help the robot system eliminate accumulated errors and a safe human-like elevator button manipulation.