Weed image augmentation by ControlNet-added stable diffusion for multi-class weed detection

Robust weed recognition for vision-guided weeding relies on curating large-scale, diverse field datasets, which however are practically difficult to come by. Text-to-image generative artificial intelligence opens new avenues for synthesizing perceptually realistic images beneficial for wide-ranging computer vision tasks in precision agriculture. This study investigates the efficacy of state-of-the-art diffusion models as an image augmentation technique for synthesizing multi-class weed images towards enhanced weed detection performance. A three-season 10-weed-class dataset was created as a testbed for image generation and weed detection tasks. The ControlNet-added stable diffusion models were trained to generate weed images with broad intra-class variations of targeted weed species and diverse backgrounds to adapt to changing field conditions. The quality of generated images was assessed using metrics including the Fréchet Inception Distance (FID) and Inception Score (IS), resulting in an average FID of 0.98 and IS of 3.63. The generated weed images were selected to supplement real-world images for weed detection by YOLOv8-large. Combining the manually selected, generated images with real images yielded an overall mAP@50:95 of 88.3 % and mAP@50 of 95.0 %, representing performance gains of 1.4 % and 0.8 %, respectively, compared to the baseline model trained using only real images. It also performed competitively or comparably with modeling by combining real images with the images generated by external, traditional data augmentation techniques. The proposed automated post-generation image filtering approach still needs improvements to select high-quality images for enhanced weed detection. Both the weed dataset¹ and software programs² developed in this study have been made publicly available. Considerable research is needed to exploit more controllable diffusion models for generating high-fidelity, diverse weed images to substantially enhance weed detection in changing field conditions.