LVF: A language and vision fusion framework for tomato diseases segmentation

Abstract

With the development of deep learning technology, the control of tomato diseases has emerged as a crucial aspect of intelligent agricultural management. While current research on tomato disease segmentation has made considerable strides, challenges persist due to the susceptibility of tomato leaf diseases to strong light reflections and shadow gradients in sunlight. Additionally, the complex backgrounds found in agricultural fields often lead to model confusion, resulting in inaccurate segmentation. Traditional methods for tomato disease segmentation rely on single-modal image-based models, which struggle when dealing with the nuanced features and limited scope of tomato leaf diseases. To address these issues, our study introduces the LVF framework, a dual-modal approach combining image and text information for pre-segmentation of tomato diseases. We began by creating a new dataset labeled with both images and text, specifically focusing on diseased tomato leaves with guidance from agricultural experts. For image processing, we developed a probabilistic differential fusion network to mitigate interference caused by high-frequency noise, leveraging color and grayscale images. Furthermore, our reinforcement feature network and threshold filtering network enhance useful information while filtering out negative information from the fused images. In text processing, we proposed a multi-scale cross-nesting network to integrate semantic information about diseases across different scales and types. By nesting Bert-processed word vectors with fused image vectors, our model gains a deeper understanding of semantic information, thereby improving its ability to segment crop diseases accurately. Our experiments, conducted on self-constructed tomato datasets as well as public datasets for tomatoes and maize, demonstrated the efficacy and robustness of our approach in leaf disease segmentation. The LVF framework offers a valuable tool to enhance the accuracy of crop disease segmentation, especially in complex agricultural environments.