Real-Time Object Coordinate Detection and Manipulator Control Using Rigidly Trained Convolutional Neural Networks

Objects embedded in the environment, such as switches, control buttons, sockets, et al., are devices that need frequent operations. To devise manipulators to operate such devices automatically, we propose a visual-position control scheme that directly converts the visual coordinate detections to motor commands. We train ConvNets with rigid 3D coordinate information, which is obtained from a single basis image of the target object. Our proposed training data preparation frameworks automatically generate and organize the required structure of the training images for the network. The ConvNet’s superior image recognition capability results in high success rate in object detection and high precision in coordinate estimation. In our static experiments, in-range plane coordinate detection achieves an average success rate of 91% from various view-point directions; the depth coordinate detection achieves an average success rate of 86% based on an extended success range. In our dynamic experiments, a low-precision manipulator was used to press a down elevator call button and achieved an overall success rate of 98%. A high-precision manipulator was used for an object localization task and achieved a precision of ± 0.3 mm using a low-resolution camera.