Autonomous smart palm tree harvesting with deep learning-enabled date fruit type and maturity stage classification

Abstract

This work proposes an innovative autonomous system based on intelligent deep-transfer learning for the sustainable harvesting of palm trees. The machine learning-based autonomous robot detects and captures the date fruit bunches on palm trees in the natural farm environment using the lightweight you only look once (YOLO)v8 algorithm. Five different types of fruit bunches are further classified using a deep transfer learning system based on the type (Khalas, Barhi, Sullaj, Meneifi, and Naboot Saif) and the maturity stage of date fruit (Immature, Khalal, Khalal with Rutab, Pre-Tamar, and Tamar) for their efficient, faster, and accurate harvesting. Five deep convolutional neural network (CNN) models, the Alex Krizhevsky network (AlexNet), the visual geometry group (VGG-16), the residual network (ResNet-50), Inception-v3 and Efficient Net, were trained on around 12,000 images at the bunch level for the two classification tasks. The findings of various performed experiments suggested that the VGG-16 network outperforms the compared models with maximum achieved testing accuracies of 98.89% and 98.17% for date type and maturity stage classification, respectively. The obtained testing accuracies of AlexNet, ResNet-50, Efficient Net, and Inception-v3 models are 97.33%, 97.87%, 98.39%, 96.61% 98%, 93%, and 86.5% for both date type/maturity stage predictions. These obtained accuracies are superior than the state-of-the-art legacy models. Autonomous robotic vehicle front and top cameras are used to localize the quick response (QR)-labeled palm trees using canny edge detection and hough transformation, and date bunch detection and capturing using the trained YOLOv8 algorithm. The robotic vehicle transfers all captured images using Firebase after the completion of the farm journey. The developed and integrated front-end user interface (UI) provides ease to farmers for the two classification tasks of the retrieved images, along with the harvesting decision for each image. The use of proposed sustainable smart harvesting robots to classify and analyze date bunches in the natural environment can significantly improve the yield and global supply chain of this fruit.