Yield estimation in precision viticulture by combining deep segmentation and depth-based clustering

Abstract

Grapevine phenotyping, that is the process of determining the physical properties (e.g., size, shape, and number) of grape bunches, provides valuable information for growth and health monitoring, yield estimation and efficient crop management in precision viticulture. Currently, grape bunch counting and sizing is done manually, which is labor intensive and often impractical for large-scale field applications. This paper describes a novel framework to automatically detect, count and estimate the volume/weight of grape bunches using RGB and depth data acquired in the field by a farmer robot. The proposed pipeline starts with the semantic segmentation of RGB images based on a pre-trained MANet architecture with EfficientnetB3 backbone to separate fruit from non-fruit regions. The segmented fruit mask is then projected onto the co-registered depth image to recover a depth mask, allowing for three-dimensional (3D) data association. After a pre-processing step to correct anomalies, such as corrupted and missing values, and to remove outliers, a depth gradient-based clustering algorithm is applied that detects individual grape bunch clusters. This enables the separation of adjacent and partially overlapping bunches. In addition, a method to reconstruct the whole 3D shape of a bunch is introduced, so as to provide an estimate of volume and weight. Experiments performed in a commercial vineyard in Italy are presented showing that, despite the low quality and high variability of the input images, the proposed approach is able to count grape bunch clusters with an average error of about 12% with respect to visual ground-truth and an average error less than 30% with respect to manual weight measurements. It is also shown that the processing framework can be applied to geo-referenced image sequences acquired by the farmer robot while traversing vineyard rows, thus providing an automated pipeline for the generation of high-resolution yield maps for precision viticulture applications.