A novel self-supervised method for in-field occluded apple ripeness determination

Abstract

The full view of the apples in the orchard is often obscured by leaves and trunks, making it challenging to accurately determine their ripeness, whilst it is an essential yet difficult task for apple-harvesting robots. Within this context, we propose a novel method to address two critical challenges: ripeness determination and in-field occlusion. The proposed method is trained in a self-supervised manner on a dataset consisting of less than 1% labelled images and the rest of unlabelled images. It is made up of three key parts: a reconstructor, a feature extractor, and a predictor. The reconstructor is designed to reconstruct the missing parts of occluded apples. The feature extractor is introduced to learn ripeness-related features from the vast number of unlabelled images. Unlike the previous approaches classifying the fruit ripeness into several discrete categories, the predictor uses the learned features to generate a continuous ripeness score in the range between 0.0 and 1.0, thus eliminating the need to subjectively pre-define ripeness stages and offering end-users the flexibility to make their own decisions.

Experimental results comparing our method to another method with different settings show that our method achieves the best Structural Similarity Index Measure (SSIM) of 0.75 and the second-best Peak-Signal-to-Noise Ratio (PSNR) of 25.36 for reconstructing missing apple parts, whilst using the fewest 86.3M parameters. Besides, our method outperforms 15 other self-supervised methods and even a supervised method in the ripeness score prediction, with the smallest score 0.0127 for fully unripe and the highest score 0.8933 for fully ripe apples. The results demonstrate the potential of our method to be incorporated with in-field robotic systems, enabling them to assess ripeness for selective harvesting effectively. It is helpful to monitor the overall ripeness of large orchards digitally, aid the decision-making processes and advance the goals of smart and precision agriculture.