Selective fruit harvesting prediction and 6D pose estimation based on YOLOv7 multi-parameter recognition

Abstract

For crops that need to be harvested in batches based on maturity, the harvesting operation needs to select individual fruits that have developed and matured for harvest. Therefore, The real-time performance and accuracy of fruit target recognition and localization tasks in selective harvesting robotic operations play a crucial role in the improvement of harvesting efficiency. Unlike the spatial localization of fruits, the estimation of the 6D pose of fruits, which requires more parameters, negatively impacts the network’s real-time and generalization. Therefore, this study proposes selective harvest recognition and a 6D pose estimation algorithm based on YOLOv7 multi-parameter recognition using cucumber as the research object. First, the YOLOV7-hv algorithm is proposed by improving the structure of the YOLOv7 network, adding the key points recognition branch and the mask generation branch to identify the suitable fruit targets for harvesting, and realizing the pose key points recognition and the instance segmentation of the suitable fruit targets. Based on the multinomial parameters recognized by the YOLOV7-hv network, the YOLOv7-hv picking 6D pose estimation algorithm is proposed to realize the estimation of fruit picking 6D pose. In the cucumber fruit datasets captured in this study, the YOLOV7-hv algorithm has AP of 94.0 % for target detection, OKS of 0.882 for key points detection, mIOU of 93.8 % for the segmentation task, Fps of 43 for the overall network，the mean position error of 6.18 mm for localization, the average time consumption of 8.2 ms for localization, the angular error of 6.25° for pose estimation and the average time consumption of 8.4 ms for pose estimation. In embedded devices, the model still maintains close evaluation metrics and good real-time performance. The various performance metrics indicate that the method proposed in this paper enables real-time and accurate recognition and pose estimation of fruit targets.