George A Neyarapally, Leihong Wu, Joshua Xu, Esther H Zhou, Oanh Dang, Joann Lee, Dharmang Mehta, Rochelle D Vaughn, Ellen Pinnow, Hong Fang
{"title":"描述并验证用于识别 FDA 标签中不良事件变更的新型人工智能工具 LabelComp。","authors":"George A Neyarapally, Leihong Wu, Joshua Xu, Esther H Zhou, Oanh Dang, Joann Lee, Dharmang Mehta, Rochelle D Vaughn, Ellen Pinnow, Hong Fang","doi":"10.1007/s40264-024-01468-8","DOIUrl":null,"url":null,"abstract":"<p><strong>Introduction: </strong>The accurate identification and timely updating of adverse reactions in drug labeling are crucial for patient safety and effective drug use. Postmarketing surveillance plays a pivotal role in identifying previously undetected adverse events (AEs) that emerge when a drug is used in broader and more diverse patient populations. However, traditional methods of updating drug labeling with new AE information have been manual, time consuming, and error prone. This paper introduces the LabelComp tool, an innovative artificial intelligence (AI) tool designed to enhance the efficiency and accuracy of postmarketing drug safety surveillance. Utilizing a combination of text analytics and a trained Bidirectional Encoder Representations from Transformers (BERT) model, the LabelComp tool automatically identifies changes in AE terms from updated drug labeling documents.</p><p><strong>Objective: </strong>Our objective was to create and validate an AI tool with high accuracy that could enable researchers and FDA reviewers to efficiently identify safety-related drug labeling changes.</p><p><strong>Results: </strong>Our validation study of 87 drug labeling PDF pairs demonstrates the tool's high accuracy, with F1 scores of overall performance ranging from 0.795 to 0.936 across different evaluation tiers and a recall of at least 0.997 with only one missed AE out of 483 total AEs detected, indicating the tool's efficacy in identifying new AEs.</p><p><strong>Conclusion: </strong>The LabelComp tool can support drug safety surveillance and inform regulatory decision-making. The publication of this tool also aims to encourage further community-driven enhancements, aligning with broader interests in applying AI to advance regulatory science and public health.</p>","PeriodicalId":11382,"journal":{"name":"Drug Safety","volume":" ","pages":"1265-1274"},"PeriodicalIF":4.0000,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11554693/pdf/","citationCount":"0","resultStr":"{\"title\":\"Description and Validation of a Novel AI Tool, LabelComp, for the Identification of Adverse Event Changes in FDA Labeling.\",\"authors\":\"George A Neyarapally, Leihong Wu, Joshua Xu, Esther H Zhou, Oanh Dang, Joann Lee, Dharmang Mehta, Rochelle D Vaughn, Ellen Pinnow, Hong Fang\",\"doi\":\"10.1007/s40264-024-01468-8\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Introduction: </strong>The accurate identification and timely updating of adverse reactions in drug labeling are crucial for patient safety and effective drug use. Postmarketing surveillance plays a pivotal role in identifying previously undetected adverse events (AEs) that emerge when a drug is used in broader and more diverse patient populations. However, traditional methods of updating drug labeling with new AE information have been manual, time consuming, and error prone. This paper introduces the LabelComp tool, an innovative artificial intelligence (AI) tool designed to enhance the efficiency and accuracy of postmarketing drug safety surveillance. Utilizing a combination of text analytics and a trained Bidirectional Encoder Representations from Transformers (BERT) model, the LabelComp tool automatically identifies changes in AE terms from updated drug labeling documents.</p><p><strong>Objective: </strong>Our objective was to create and validate an AI tool with high accuracy that could enable researchers and FDA reviewers to efficiently identify safety-related drug labeling changes.</p><p><strong>Results: </strong>Our validation study of 87 drug labeling PDF pairs demonstrates the tool's high accuracy, with F1 scores of overall performance ranging from 0.795 to 0.936 across different evaluation tiers and a recall of at least 0.997 with only one missed AE out of 483 total AEs detected, indicating the tool's efficacy in identifying new AEs.</p><p><strong>Conclusion: </strong>The LabelComp tool can support drug safety surveillance and inform regulatory decision-making. The publication of this tool also aims to encourage further community-driven enhancements, aligning with broader interests in applying AI to advance regulatory science and public health.</p>\",\"PeriodicalId\":11382,\"journal\":{\"name\":\"Drug Safety\",\"volume\":\" \",\"pages\":\"1265-1274\"},\"PeriodicalIF\":4.0000,\"publicationDate\":\"2024-12-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11554693/pdf/\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Drug Safety\",\"FirstCategoryId\":\"88\",\"ListUrlMain\":\"https://doi.org/10.1007/s40264-024-01468-8\",\"RegionNum\":2,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2024/7/31 0:00:00\",\"PubModel\":\"Epub\",\"JCR\":\"Q1\",\"JCRName\":\"PHARMACOLOGY & PHARMACY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Drug Safety","FirstCategoryId":"88","ListUrlMain":"https://doi.org/10.1007/s40264-024-01468-8","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/7/31 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"PHARMACOLOGY & PHARMACY","Score":null,"Total":0}
Description and Validation of a Novel AI Tool, LabelComp, for the Identification of Adverse Event Changes in FDA Labeling.
Introduction: The accurate identification and timely updating of adverse reactions in drug labeling are crucial for patient safety and effective drug use. Postmarketing surveillance plays a pivotal role in identifying previously undetected adverse events (AEs) that emerge when a drug is used in broader and more diverse patient populations. However, traditional methods of updating drug labeling with new AE information have been manual, time consuming, and error prone. This paper introduces the LabelComp tool, an innovative artificial intelligence (AI) tool designed to enhance the efficiency and accuracy of postmarketing drug safety surveillance. Utilizing a combination of text analytics and a trained Bidirectional Encoder Representations from Transformers (BERT) model, the LabelComp tool automatically identifies changes in AE terms from updated drug labeling documents.
Objective: Our objective was to create and validate an AI tool with high accuracy that could enable researchers and FDA reviewers to efficiently identify safety-related drug labeling changes.
Results: Our validation study of 87 drug labeling PDF pairs demonstrates the tool's high accuracy, with F1 scores of overall performance ranging from 0.795 to 0.936 across different evaluation tiers and a recall of at least 0.997 with only one missed AE out of 483 total AEs detected, indicating the tool's efficacy in identifying new AEs.
Conclusion: The LabelComp tool can support drug safety surveillance and inform regulatory decision-making. The publication of this tool also aims to encourage further community-driven enhancements, aligning with broader interests in applying AI to advance regulatory science and public health.
期刊介绍:
Drug Safety is the official journal of the International Society of Pharmacovigilance. The journal includes:
Overviews of contentious or emerging issues.
Comprehensive narrative reviews that provide an authoritative source of information on epidemiology, clinical features, prevention and management of adverse effects of individual drugs and drug classes.
In-depth benefit-risk assessment of adverse effect and efficacy data for a drug in a defined therapeutic area.
Systematic reviews (with or without meta-analyses) that collate empirical evidence to answer a specific research question, using explicit, systematic methods as outlined by the PRISMA statement.
Original research articles reporting the results of well-designed studies in disciplines such as pharmacoepidemiology, pharmacovigilance, pharmacology and toxicology, and pharmacogenomics.
Editorials and commentaries on topical issues.
Additional digital features (including animated abstracts, video abstracts, slide decks, audio slides, instructional videos, infographics, podcasts and animations) can be published with articles; these are designed to increase the visibility, readership and educational value of the journal’s content. In addition, articles published in Drug Safety Drugs may be accompanied by plain language summaries to assist readers who have some knowledge of, but not in-depth expertise in, the area to understand important medical advances.