Self-supervised learning-based multi-source spectral fusion for fruit quality evaluation:a case study in mango fruit ripeness prediction

Rapid and non-destructive techniques for fruit quality evaluation are widely concerned in modern agro-industry. Spectroscopy is one of the most commonly used techniques in this field. With the growing popularity of various spectroscopic instruments, it is indeed worthwhile to explore modeling with multi-source spectral data to achieve more accurate predictions. Nonetheless, a major challenge is acquiring enough labeled samples, as measuring fruit chemical values is laborious, expensive, and time-consuming, which hinders the development of a reliable prediction model. Therefore, this study aims to develop a model for predicting the internal chemical composition of fruits by integrating multi-source spectral fusion combined with self-supervised learning (SSL). A visible (Vis) and near-infrared (NIR) spectral dataset related to dry matter content (DMC) prediction in mango fruit is used as an example to validate the effectiveness of the proposed method. To obtain multi-source spectral data, the Vis and NIR portions are processed as two separate spectral ranges. An SSL pre-training is performed utilizing a large amount of raw unlabeled spectral data to extract general knowledge, which is subsequently migrated to a downstream task for fine-tuning. The experimental results indicate that the multi-source spectral fusion model performs better than the single-source spectral model. Moreover, SSL solves the data scarcity problem and outperforms non-pre-trained models in downstream DMC prediction tasks with less computational overhead. Remarkably, utilizing only less than 10% of the total samples is sufficient to achieve a performance close to 99% of the best results. The presented method has great potential in spectral analysis of food and agro-products.