A customized density map model and segment anything model for cotton boll number, size, and yield prediction in aerial images

Abstract

The number of cotton bolls is an important phenotyping trait not only for breeders but also for growers. It can provide information on the physiological and genetic mechanisms of plant growth and aid decision-making in crop management. However, traditional visual inspection in the field is time-consuming and laborious. With the application of drones in the agricultural domain, there is promising potential to collect data expediently. In this paper, we integrated the improved Distribution Matching for crowd Counting (DM-Count) and Segment Anything Model (SAM) to predict cotton boll number, size, and yield in aerial images. The cotton plots were first extracted from the raw aerial images using boundaries derived from orthophotos. Then, a convolutional neural network (DM-Count) was introduced as a baseline and customized by replacing the VGG19 backbone and adding a pixel loss. The customized network was first pretrained on ground images and then fine-tuned on aerial images to predict the density map, where the number and locations of cotton bolls can be obtained. The zero-shot foundation model SAM was investigated to segment cotton bolls with the point prompts provided by customized DM-Count. The respective numbers of bolls and segmented pixels were compared for seed cotton yield estimation. The experimental results showed that the customized model obtained a mean absolute error (MAE) of 1.78 per square meter and a mean absolute percentage error (MAPE) of 4.39 % on the testing dataset, with a high correlation between the predicted boll number and ground truth (R² = 0.91). The AP50 of SAM for cotton boll segmentation was 0.63. The segmented masks were used to delineate the boll size differences among the four genotypes, and it was found that the average boll size of Pima was 452 pixels, which was significantly smaller than Acala Maxxa, UA 48 and Tamcot Sphinx. Moreover, the yield estimation using the boll number was better than that using the pixel number, with an R² = 0.70. Combing the boll number and the pixel number can achieve a slightly higher R² of 0.72 for yield estimation. Overall, the customized model can count cotton bolls in aerial images accurately and estimate seed cotton yield effectively, which could significantly benefit breeders in developing genotypes with high yields, as well as help growers in yield estimation and crop management.