Mojdeh Saadati, Aditya Balu, Shivani Chiranjeevi, Talukder Zaki Jubery, Asheesh K Singh, Soumik Sarkar, Arti Singh, Baskar Ganapathysubramanian
{"title":"用于昆虫稳健分类的分布外检测算法","authors":"Mojdeh Saadati, Aditya Balu, Shivani Chiranjeevi, Talukder Zaki Jubery, Asheesh K Singh, Soumik Sarkar, Arti Singh, Baskar Ganapathysubramanian","doi":"10.34133/plantphenomics.0170","DOIUrl":null,"url":null,"abstract":"<p><p>Plants encounter a variety of beneficial and harmful insects during their growth cycle. Accurate identification (i.e., detecting insects' presence) and classification (i.e., determining the type or class) of these insect species is critical for implementing prompt and suitable mitigation strategies. Such timely actions carry substantial economic and environmental implications. Deep learning-based approaches have produced models with good insect classification accuracy. Researchers aim to implement identification and classification models in agriculture, facing challenges when input images markedly deviate from the training distribution (e.g., images like vehicles, humans, or a blurred image or insect class that is not yet trained on). Out-of-distribution (OOD) detection algorithms provide an exciting avenue to overcome these challenges as they ensure that a model abstains from making incorrect classification predictions on images that belong to non-insect and/or untrained insect classes. As far as we know, no prior in-depth exploration has been conducted on the role of the OOD detection algorithms in addressing agricultural issues. Here, we generate and evaluate the performance of state-of-the-art OOD algorithms on insect detection classifiers. These algorithms represent a diversity of methods for addressing an OOD problem. Specifically, we focus on extrusive algorithms, i.e., algorithms that wrap around a well-trained classifier without the need for additional co-training. We compared three OOD detection algorithms: (a) maximum softmax probability, which uses the softmax value as a confidence score; (b) Mahalanobis distance (MAH)-based algorithm, which uses a generative classification approach; and (c) energy-based algorithm, which maps the input data to a scalar value, called energy. We performed an extensive series of evaluations of these OOD algorithms across three performance axes: (a) Base model accuracy: How does the accuracy of the classifier impact OOD performance? (b) How does the level of dissimilarity to the domain impact OOD performance? (c) Data imbalance: How sensitive is OOD performance to the imbalance in per-class sample size? Evaluating OOD algorithms across these performance axes provides practical guidelines to ensure the robust performance of well-trained models in the wild, which is a key consideration for agricultural applications. Based on this analysis, we proposed the most effective OOD algorithm as wrapper for the insect classifier with highest accuracy. We presented the results of its OOD detection performance in the paper. Our results indicate that OOD detection algorithms can significantly enhance user trust in insect pest classification by abstaining classification under uncertain conditions.</p>","PeriodicalId":20318,"journal":{"name":"Plant Phenomics","volume":null,"pages":null},"PeriodicalIF":7.6000,"publicationDate":"2024-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11065417/pdf/","citationCount":"0","resultStr":"{\"title\":\"Out-of-Distribution Detection Algorithms for Robust Insect Classification.\",\"authors\":\"Mojdeh Saadati, Aditya Balu, Shivani Chiranjeevi, Talukder Zaki Jubery, Asheesh K Singh, Soumik Sarkar, Arti Singh, Baskar Ganapathysubramanian\",\"doi\":\"10.34133/plantphenomics.0170\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><p>Plants encounter a variety of beneficial and harmful insects during their growth cycle. Accurate identification (i.e., detecting insects' presence) and classification (i.e., determining the type or class) of these insect species is critical for implementing prompt and suitable mitigation strategies. Such timely actions carry substantial economic and environmental implications. Deep learning-based approaches have produced models with good insect classification accuracy. Researchers aim to implement identification and classification models in agriculture, facing challenges when input images markedly deviate from the training distribution (e.g., images like vehicles, humans, or a blurred image or insect class that is not yet trained on). Out-of-distribution (OOD) detection algorithms provide an exciting avenue to overcome these challenges as they ensure that a model abstains from making incorrect classification predictions on images that belong to non-insect and/or untrained insect classes. As far as we know, no prior in-depth exploration has been conducted on the role of the OOD detection algorithms in addressing agricultural issues. Here, we generate and evaluate the performance of state-of-the-art OOD algorithms on insect detection classifiers. These algorithms represent a diversity of methods for addressing an OOD problem. Specifically, we focus on extrusive algorithms, i.e., algorithms that wrap around a well-trained classifier without the need for additional co-training. We compared three OOD detection algorithms: (a) maximum softmax probability, which uses the softmax value as a confidence score; (b) Mahalanobis distance (MAH)-based algorithm, which uses a generative classification approach; and (c) energy-based algorithm, which maps the input data to a scalar value, called energy. We performed an extensive series of evaluations of these OOD algorithms across three performance axes: (a) Base model accuracy: How does the accuracy of the classifier impact OOD performance? (b) How does the level of dissimilarity to the domain impact OOD performance? (c) Data imbalance: How sensitive is OOD performance to the imbalance in per-class sample size? Evaluating OOD algorithms across these performance axes provides practical guidelines to ensure the robust performance of well-trained models in the wild, which is a key consideration for agricultural applications. Based on this analysis, we proposed the most effective OOD algorithm as wrapper for the insect classifier with highest accuracy. We presented the results of its OOD detection performance in the paper. Our results indicate that OOD detection algorithms can significantly enhance user trust in insect pest classification by abstaining classification under uncertain conditions.</p>\",\"PeriodicalId\":20318,\"journal\":{\"name\":\"Plant Phenomics\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":7.6000,\"publicationDate\":\"2024-04-30\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11065417/pdf/\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Plant Phenomics\",\"FirstCategoryId\":\"97\",\"ListUrlMain\":\"https://doi.org/10.34133/plantphenomics.0170\",\"RegionNum\":1,\"RegionCategory\":\"农林科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2024/1/1 0:00:00\",\"PubModel\":\"eCollection\",\"JCR\":\"Q1\",\"JCRName\":\"AGRONOMY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Plant Phenomics","FirstCategoryId":"97","ListUrlMain":"https://doi.org/10.34133/plantphenomics.0170","RegionNum":1,"RegionCategory":"农林科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/1/1 0:00:00","PubModel":"eCollection","JCR":"Q1","JCRName":"AGRONOMY","Score":null,"Total":0}
Out-of-Distribution Detection Algorithms for Robust Insect Classification.
Plants encounter a variety of beneficial and harmful insects during their growth cycle. Accurate identification (i.e., detecting insects' presence) and classification (i.e., determining the type or class) of these insect species is critical for implementing prompt and suitable mitigation strategies. Such timely actions carry substantial economic and environmental implications. Deep learning-based approaches have produced models with good insect classification accuracy. Researchers aim to implement identification and classification models in agriculture, facing challenges when input images markedly deviate from the training distribution (e.g., images like vehicles, humans, or a blurred image or insect class that is not yet trained on). Out-of-distribution (OOD) detection algorithms provide an exciting avenue to overcome these challenges as they ensure that a model abstains from making incorrect classification predictions on images that belong to non-insect and/or untrained insect classes. As far as we know, no prior in-depth exploration has been conducted on the role of the OOD detection algorithms in addressing agricultural issues. Here, we generate and evaluate the performance of state-of-the-art OOD algorithms on insect detection classifiers. These algorithms represent a diversity of methods for addressing an OOD problem. Specifically, we focus on extrusive algorithms, i.e., algorithms that wrap around a well-trained classifier without the need for additional co-training. We compared three OOD detection algorithms: (a) maximum softmax probability, which uses the softmax value as a confidence score; (b) Mahalanobis distance (MAH)-based algorithm, which uses a generative classification approach; and (c) energy-based algorithm, which maps the input data to a scalar value, called energy. We performed an extensive series of evaluations of these OOD algorithms across three performance axes: (a) Base model accuracy: How does the accuracy of the classifier impact OOD performance? (b) How does the level of dissimilarity to the domain impact OOD performance? (c) Data imbalance: How sensitive is OOD performance to the imbalance in per-class sample size? Evaluating OOD algorithms across these performance axes provides practical guidelines to ensure the robust performance of well-trained models in the wild, which is a key consideration for agricultural applications. Based on this analysis, we proposed the most effective OOD algorithm as wrapper for the insect classifier with highest accuracy. We presented the results of its OOD detection performance in the paper. Our results indicate that OOD detection algorithms can significantly enhance user trust in insect pest classification by abstaining classification under uncertain conditions.
期刊介绍:
Plant Phenomics is an Open Access journal published in affiliation with the State Key Laboratory of Crop Genetics & Germplasm Enhancement, Nanjing Agricultural University (NAU) and published by the American Association for the Advancement of Science (AAAS). Like all partners participating in the Science Partner Journal program, Plant Phenomics is editorially independent from the Science family of journals.
The mission of Plant Phenomics is to publish novel research that will advance all aspects of plant phenotyping from the cell to the plant population levels using innovative combinations of sensor systems and data analytics. Plant Phenomics aims also to connect phenomics to other science domains, such as genomics, genetics, physiology, molecular biology, bioinformatics, statistics, mathematics, and computer sciences. Plant Phenomics should thus contribute to advance plant sciences and agriculture/forestry/horticulture by addressing key scientific challenges in the area of plant phenomics.
The scope of the journal covers the latest technologies in plant phenotyping for data acquisition, data management, data interpretation, modeling, and their practical applications for crop cultivation, plant breeding, forestry, horticulture, ecology, and other plant-related domains.