Improving long-tailed pest classification using diffusion model-based data augmentation

Abstract

Long-tail problem is common in large-scale agricultural datasets, posing significant challenges to agricultural research. It is often resulting from the prohibitively high costs of data collection, the challenges of obtaining accurate, comprehensive data, and the restricted access to diverse sources of information. This issue manifests especially within agricultural pest datasets, where the imbalance in the frequency of different pest types can severely hinder detection accuracy. To counteract this pervasive challenge, this paper introduces a robust method leveraging the power of a diffusion model to address the long-tailed problem effectively. Our method focuses on fine-tuning specialized pre-trained models to generate highly realistic pest images, providing a critical solution for balancing the dataset’s distribution. This paper also presents a visualization technique that offers a clear, intuitive representation of the long-tailed problem’s impact on the dataset. By producing high-quality synthetic images using the diffusion model, our method not only balances the uneven data distribution but also reduces the discrepancies between real and synthetic data, effectively mitigating the under-representation of tail categories. The experimental results, tested on the widely-used IP102 large-scale pest dataset, confirm the superiority of our approach. The method strikes an optimal balance between sample fidelity and diversity, outperforming traditional methods in image quality. Moreover, it demonstrates remarkable performance in pest classification tasks, achieving the highest evaluation metrics and showcasing its ability to address the long-tailed problem with notable success.