Disease and pest diagnosis plays a critical role in managing and controlling the damage caused by plant diseases and pests. This study employs a content-based image retrieval approach to diagnose diseases and pests, suggesting similar candidate images to assist in decision-making. Previous research in disease and pest diagnosis has relied on large models for feature extraction, posing challenges for deployment in resource-constrained environments like mobile devices. To address these challenges, this study proposes a lightweight feature extraction model, Shuffle-PG, which integrates the computationally efficient ShuffleNet v2 model with pointwise group convolution. Additionally, a method for fine-tuning the feature extraction model using deep metric learning based on contrastive loss was developed to enhance discriminative feature extraction. To validate the effectiveness of the proposed method, experiments were conducted using plant disease and pest datasets specifically collected for this study. The results show that the proposed Shuffle-PG model uses approximately 20 times fewer parameters and reduces computational costs by an order of magnitude compared to existing benchmark models, while achieving higher mean average precision scores of 97.7 % and 98.8 % for the disease and pest datasets, respectively.