Strawberry harvest date prediction using multi-feature fusion deep learning in plant factory

Abstract

Strawberries have high consumer demand due to their palatability and nutritional benefits. Commercial strawberry production in plant factories with artificial lighting (PFALs) is gaining popularity as a viable strategy for improving economic viability through high-quality fruit production. Accurate information of the optimal harvest date is crucial for optimizing harvesting decisions. While numerous studies utilize deep learning to assess strawberry ripeness, they typically only categorize generalized ripeness levels instead of predicting specific harvest dates, leaving a gap with the practical needs of growers. In this study, we proposed a two-stage multi-feature fusion model for strawberry harvest date prediction and integrated it with a web application to facilitate practical production management in PFALs. The model consists of a fruit segmentation network and a ripeness prediction network. A time-series image dataset of single fruits was constructed to continuously track the ripening process of strawberries, and a five-stage division of strawberry ripeness stages depending on optimal harvest dates was defined. A U-Net based segmentation network with post-processing was developed to automatically extract only the target fruits, which showed a reliable performance with a mIoU of 0.977. A multi-feature fusion network called Triple-Branch Attention Fusion (TBAF) was built to predict the ripeness categories with information on optimal harvest dates. The results showed that the TBAF model with fusion of color, attention-enhanced, and low-level shape features exhibited the highest performance compared to baseline models, with an overall accuracy of 0.859 and an F1 score of 0.859. In addition, a user-friendly web application was developed with the deployment of deep learning models and inspection video processing workflow to support strawberry harvesting in PFALs. Overall, this study demonstrated a prototype approach utilizing deep learning to provide essential information for grower’s decision making in practical strawberry production.