An improved binocular localization method for apple based on fruit detection using deep learning

Apple picking robot is now being developed as an alternative to hand picking due to a great demand for labor during apple harvest season. Accurate detection and localization of target fruit is necessary for robotic apple picking. Detection accuracy has a great influence on localization results. Although current researches on detection and localization of apples using traditional image algorithms can obtain good results under laboratory conditions, it is difficult to accurately detect and locate objects in natural field with complex environments. With the rapid development of artificial intelligence, accuracy of apple detection based on deep learning has been significantly improved. Therefore, a deep learning-based method was developed to accurately detect and locate the position of fruit. For different localization methods, binocular localization is a widely used localization method for its bionic principle and lower equipment cost. Hence, this paper proposed an improved binocular localization method for apple based on fruit detection using deep learning. First, apples of binocular images were detected by Faster R-CNN. After that, a segmentation based on chromatic aberration and chromatic aberration ratio was applied to segment apple and background pixels in bounding box of detected fruit. Furthermore, template matching with parallel polar line constraint was used to match apples in left and right images. Finally, two feature points on apples were selected to directly calculate three dimensional coordinates of feature points with the binocular localization principle. In this study, Faster R-CNN achieved an AP of 88.12% with an average detection speed of 0.32 s for an image. Meanwhile, standard deviation and localization precision of depth of two feature points on each apple were calculated to evaluate localization. Results showed that the average standard deviation and the average localization precision of 76 groups of datasets were 0.51 cm and 99.64%, respectively. Results indicated that the proposed improved binocular localization method is promising for fruit localization.