Pub Date : 2024-12-01DOI: 10.1016/j.mcpdig.2024.07.005
Curtis P. Langlotz MD, PhD , Johanna Kim MPH, MBA , Nigam Shah MBBS, PhD , Matthew P. Lungren MD, MPH , David B. Larson MD, MBA , Somalee Datta PhD , Fei Fei Li PhD , Ruth O’Hara PhD , Thomas J. Montine MD, PhD , Robert A. Harrington MD , Garry E. Gold MD, MS
Artificial intelligence (AI) and machine learning (ML) are driving innovation in biosciences and are already affecting key elements of medical scholarship and clinical care. Many schools of medicine are capitalizing on the promise of these new technologies by establishing academic units to catalyze and grow research and innovation in AI/ML. At Stanford University, we have developed a successful model for an AI/ML research center with support from academic leaders, clinical departments, extramural grants, and industry partners. The Center for Artificial Intelligence in Medicine and Imaging uses the following 4 key tactics to support AI/ML research: project-based learning opportunities that build interdisciplinary collaboration; internal grant programs that catalyze extramural funding; infrastructure that facilitates the rapid creation of large multimodal AI-ready clinical data sets; and educational and open data programs that engage the broader research community. The center is based on the premise that foundational and applied research are not in tension but instead are complementary. Solving important biomedical problems with AI/ML requires high-quality foundational team science that incorporates the knowledge and expertise of clinicians, clinician scientists, computer scientists, and data scientists. As AI/ML becomes an essential component of research and clinical care, multidisciplinary centers of excellence in AI/ML will become a key part of the scholarly portfolio of academic medical centers and will provide a foundation for the responsible, ethical, and fair implementation of AI/ML systems.
{"title":"Developing a Research Center for Artificial Intelligence in Medicine","authors":"Curtis P. Langlotz MD, PhD , Johanna Kim MPH, MBA , Nigam Shah MBBS, PhD , Matthew P. Lungren MD, MPH , David B. Larson MD, MBA , Somalee Datta PhD , Fei Fei Li PhD , Ruth O’Hara PhD , Thomas J. Montine MD, PhD , Robert A. Harrington MD , Garry E. Gold MD, MS","doi":"10.1016/j.mcpdig.2024.07.005","DOIUrl":"10.1016/j.mcpdig.2024.07.005","url":null,"abstract":"<div><div>Artificial intelligence (AI) and machine learning (ML) are driving innovation in biosciences and are already affecting key elements of medical scholarship and clinical care. Many schools of medicine are capitalizing on the promise of these new technologies by establishing academic units to catalyze and grow research and innovation in AI/ML. At Stanford University, we have developed a successful model for an AI/ML research center with support from academic leaders, clinical departments, extramural grants, and industry partners. The Center for Artificial Intelligence in Medicine and Imaging uses the following 4 key tactics to support AI/ML research: project-based learning opportunities that build interdisciplinary collaboration; internal grant programs that catalyze extramural funding; infrastructure that facilitates the rapid creation of large multimodal AI-ready clinical data sets; and educational and open data programs that engage the broader research community. The center is based on the premise that foundational and applied research are not in tension but instead are complementary. Solving important biomedical problems with AI/ML requires high-quality foundational team science that incorporates the knowledge and expertise of clinicians, clinician scientists, computer scientists, and data scientists. As AI/ML becomes an essential component of research and clinical care, multidisciplinary centers of excellence in AI/ML will become a key part of the scholarly portfolio of academic medical centers and will provide a foundation for the responsible, ethical, and fair implementation of AI/ML systems.</div></div>","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 677-686"},"PeriodicalIF":0.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142744031","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-11-05DOI: 10.1016/j.mcpdig.2024.10.004
Janice L. Pascoe BRMP , Luqing Lu MS , Matthew M. Moore MFA , Daniel J. Blezek PhD , Annie E. Ovalle BS , Jane A. Linderbaum APRN, CNP , Matthew R. Callstrom MD, PhD , Eric E. Williamson MD
Artificial intelligence (AI) promises to revolutionize health care. Early identification of disease, appropriate test selection, and automation of repetitive tasks are expected to optimize cost-effective care delivery. However, pragmatic selection and integration of AI algorithms to enable this transformation remain challenging. Health care leaders must navigate complex decisions regarding AI deployment, considering factors such as cost of implementation, benefits to patients and providers, and institutional readiness for adoption. A successful strategy needs to align AI adoption with institutional priorities, select appropriate algorithms to be purchased or internally developed, and ensure adequate support and infrastructure. Further, successful deployment requires algorithm validation and workflow integration to ensure efficacy and usability. User-centric design principles and usability testing are critical for AI adoption, ensuring seamless integration into clinical workflows. Once deployed, continuous improvement processes and ongoing algorithm support ensure continuous benefits to the clinical practice. Vigilant planning and execution are necessary to navigate the complexities of AI implementation in the health care environment. By applying the framework outlined in this article, institutions can navigate the ever-evolving and complex environment of AI in health care to maximize the benefits of these innovative technologies.
{"title":"Strategic Considerations for Selecting Artificial Intelligence Solutions for Institutional Integration: A Single-Center Experience","authors":"Janice L. Pascoe BRMP , Luqing Lu MS , Matthew M. Moore MFA , Daniel J. Blezek PhD , Annie E. Ovalle BS , Jane A. Linderbaum APRN, CNP , Matthew R. Callstrom MD, PhD , Eric E. Williamson MD","doi":"10.1016/j.mcpdig.2024.10.004","DOIUrl":"10.1016/j.mcpdig.2024.10.004","url":null,"abstract":"<div><div>Artificial intelligence (AI) promises to revolutionize health care. Early identification of disease, appropriate test selection, and automation of repetitive tasks are expected to optimize cost-effective care delivery. However, pragmatic selection and integration of AI algorithms to enable this transformation remain challenging. Health care leaders must navigate complex decisions regarding AI deployment, considering factors such as cost of implementation, benefits to patients and providers, and institutional readiness for adoption. A successful strategy needs to align AI adoption with institutional priorities, select appropriate algorithms to be purchased or internally developed, and ensure adequate support and infrastructure. Further, successful deployment requires algorithm validation and workflow integration to ensure efficacy and usability. User-centric design principles and usability testing are critical for AI adoption, ensuring seamless integration into clinical workflows. Once deployed, continuous improvement processes and ongoing algorithm support ensure continuous benefits to the clinical practice. Vigilant planning and execution are necessary to navigate the complexities of AI implementation in the health care environment. By applying the framework outlined in this article, institutions can navigate the ever-evolving and complex environment of AI in health care to maximize the benefits of these innovative technologies.</div></div>","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 665-676"},"PeriodicalIF":0.0,"publicationDate":"2024-11-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142721288","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-31DOI: 10.1016/j.mcpdig.2024.10.005
{"title":"Reviewers for Mayo Clinic Proceedings: Digital Health (2024)","authors":"","doi":"10.1016/j.mcpdig.2024.10.005","DOIUrl":"10.1016/j.mcpdig.2024.10.005","url":null,"abstract":"","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 645-646"},"PeriodicalIF":0.0,"publicationDate":"2024-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142704497","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-30DOI: 10.1016/j.mcpdig.2024.10.003
Evelien B. van Kempen MD , Sanne E.W. Vrijlandt MD , Kelly van der Geest MSc , Sophie Lotgering MSc , Tom A. Hueting PhD , Rianne Oostenbrink MD, PhD
Clinical decision rules (CDRs) integrated into applications enhance diagnostic and treatment prediction support for clinicians, necessitating Confirmité Europeenne (CE)-mark certification to enter the European market. We describe the development of a CDR as a medical device, focusing on challenges from a physician’s perspective exemplified by the Feverkidstool (FKT), a validated CDR for febrile children. We pursued a local process, aligned with the CE-marking process, to develop the FKT as in-house developed device. We aimed to provide a blueprint for colleagues. Medical device development, conforming the medical device regulation and performed by a multidisciplinary team, encompassed 5 stages: market scan, design, production, verification and validation and conformity assessment. Regulatory processes were continuously updated. The market scan identified a need for the FKT compared with existing applications. A prototype was designed in stage 2, further adjusted and improved based on the qualitative and quantitative results of stages 2-4. Lastly, stage 5 confirmed FKT’s performance and safety. Medical device development presents challenges for physicians, requiring collaboration for technical, regulatory, and financial expertise. Multidisciplinary teamwork also poses challenges, including uncertainties regarding responsibility and timelines. After CE certification, adapting to evolving needs and ensuring data privacy highlights the ongoing nature of medical device development.
{"title":"A Blueprint for Clinical-Driven Medical Device Development: The Feverkidstool Application to Identify Children With Serious Bacterial Infection","authors":"Evelien B. van Kempen MD , Sanne E.W. Vrijlandt MD , Kelly van der Geest MSc , Sophie Lotgering MSc , Tom A. Hueting PhD , Rianne Oostenbrink MD, PhD","doi":"10.1016/j.mcpdig.2024.10.003","DOIUrl":"10.1016/j.mcpdig.2024.10.003","url":null,"abstract":"<div><div>Clinical decision rules (CDRs) integrated into applications enhance diagnostic and treatment prediction support for clinicians, necessitating Confirmité Europeenne (CE)-mark certification to enter the European market. We describe the development of a CDR as a medical device, focusing on challenges from a physician’s perspective exemplified by the Feverkidstool (FKT), a validated CDR for febrile children. We pursued a local process, aligned with the CE-marking process, to develop the FKT as in-house developed device. We aimed to provide a blueprint for colleagues. Medical device development, conforming the medical device regulation and performed by a multidisciplinary team, encompassed 5 stages: market scan, design, production, verification and validation and conformity assessment. Regulatory processes were continuously updated. The market scan identified a need for the FKT compared with existing applications. A prototype was designed in stage 2, further adjusted and improved based on the qualitative and quantitative results of stages 2-4. Lastly, stage 5 confirmed FKT’s performance and safety. Medical device development presents challenges for physicians, requiring collaboration for technical, regulatory, and financial expertise. Multidisciplinary teamwork also poses challenges, including uncertainties regarding responsibility and timelines. After CE certification, adapting to evolving needs and ensuring data privacy highlights the ongoing nature of medical device development.</div></div>","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 656-664"},"PeriodicalIF":0.0,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142703869","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-26DOI: 10.1016/j.mcpdig.2024.10.001
Viengneesee Thao PhD, MS , Ye Zhu MD, MPH, PhD , Andrew S. Tseng MD, MPH , Jonathan W. Inselman MS , Bijan J. Borah PhD , Rozalina G. McCoy MD, MS , Zachi I. Attia PhD , Francisco Lopez-Jimenez MD, MBA , Patricia A. Pellikka MD , David R. Rushlow MD, MBOE , Paul A. Friedman MD , Peter A. Noseworthy MD, MBA , Xiaoxi Yao MPH, MS, PhD
Objective
To investigate the cost-effectiveness of using artificial intelligence (AI) to screen for low ejection fraction (EF) in routine clinical practice using a pragmatic randomized controlled trial (RCT).
Patients and Methods
In a post hoc analysis of the electrocardiogram (ECG) AI-guided screening for low ejection fraction trial, we developed a decision analytic model for patients aged 18 years and older without previously diagnosed heart failure and underwent a clinically indicated ECG between August 5, 2019, and March 31, 2020. In the previously published RCT, the intervention arm underwent an AI-guided targeted screening program for low EF with a workflow embedded into routine clinical practice—AI was applied to the ECG to identify patients at high-risk and recommended clinicians to order an ECG and the control arm received usual care without the screening program. We used results from the RCT for rates of low EF diagnosis and a lifetime Markov model to project the long-term outcomes. Quality-adjusted life years (QALYs), costs of intervention and treatment, disease event costs, incremental cost-effectiveness ratio (ICER), and cost for the number needed to screen. Multiple scenario and sensitivity analyses were performed.
Results
Compared with usual care, AI-integrated ECG was cost effective, with an incremental cost-effectiveness ratio of $27,858/QALY. The program remained cost effective even with a change in patient age and follow-up time duration, although the specific ICER values varied for these parameters. The program was more cost effective in outpatient settings (ICER $1651/QALY) than in inpatient or emergency room settings.
Conclusion
Implementing AI-guided targeted screening for low EF in routine clinical practice was cost effective.
患者和方法在心电图(ECG)人工智能指导的低射血分数筛查试验的事后分析中,我们为年龄在18岁及18岁以上、既往未确诊心力衰竭且在2019年8月5日至2020年3月31日期间接受了有临床指征的心电图检查的患者开发了一个决策分析模型。在之前发表的 RCT 中,干预组接受了人工智能指导的低 EF 靶向筛查计划,并将工作流程嵌入到常规临床实践中--人工智能应用于心电图以识别高风险患者,并建议临床医生进行心电图检查;对照组接受常规护理,不接受筛查计划。我们利用 RCT 的低 EF 诊断率结果和终身马尔可夫模型来预测长期结果。结果显示了质量调整生命年 (QALY)、干预和治疗成本、疾病事件成本、增量成本效益比 (ICER) 以及筛查所需人数的成本。结果与常规护理相比,人工智能整合心电图具有成本效益,增量成本效益比为 27,858 美元/QALY。即使患者年龄和随访时间发生变化,该方案仍具有成本效益,尽管这些参数的具体 ICER 值有所不同。结论在常规临床实践中实施人工智能指导的低 EF 目标筛查具有成本效益。
{"title":"Cost-Effectiveness of Artificial Intelligence-Enabled Electrocardiograms for Early Detection of Low Ejection Fraction: A Secondary Analysis of the Electrocardiogram Artificial Intelligence-Guided Screening for Low Ejection Fraction Trial","authors":"Viengneesee Thao PhD, MS , Ye Zhu MD, MPH, PhD , Andrew S. Tseng MD, MPH , Jonathan W. Inselman MS , Bijan J. Borah PhD , Rozalina G. McCoy MD, MS , Zachi I. Attia PhD , Francisco Lopez-Jimenez MD, MBA , Patricia A. Pellikka MD , David R. Rushlow MD, MBOE , Paul A. Friedman MD , Peter A. Noseworthy MD, MBA , Xiaoxi Yao MPH, MS, PhD","doi":"10.1016/j.mcpdig.2024.10.001","DOIUrl":"10.1016/j.mcpdig.2024.10.001","url":null,"abstract":"<div><h3>Objective</h3><div>To investigate the cost-effectiveness of using artificial intelligence (AI) to screen for low ejection fraction (EF) in routine clinical practice using a pragmatic randomized controlled trial (RCT).</div></div><div><h3>Patients and Methods</h3><div>In a post hoc analysis of the electrocardiogram (ECG) AI-guided screening for low ejection fraction trial, we developed a decision analytic model for patients aged 18 years and older without previously diagnosed heart failure and underwent a clinically indicated ECG between August 5, 2019, and March 31, 2020. In the previously published RCT, the intervention arm underwent an AI-guided targeted screening program for low EF with a workflow embedded into routine clinical practice—AI was applied to the ECG to identify patients at high-risk and recommended clinicians to order an ECG and the control arm received usual care without the screening program. We used results from the RCT for rates of low EF diagnosis and a lifetime Markov model to project the long-term outcomes. Quality-adjusted life years (QALYs), costs of intervention and treatment, disease event costs, incremental cost-effectiveness ratio (ICER), and cost for the number needed to screen. Multiple scenario and sensitivity analyses were performed.</div></div><div><h3>Results</h3><div>Compared with usual care, AI-integrated ECG was cost effective, with an incremental cost-effectiveness ratio of $27,858/QALY. The program remained cost effective even with a change in patient age and follow-up time duration, although the specific ICER values varied for these parameters. The program was more cost effective in outpatient settings (ICER $1651/QALY) than in inpatient or emergency room settings.</div></div><div><h3>Conclusion</h3><div>Implementing AI-guided targeted screening for low EF in routine clinical practice was cost effective.</div></div>","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 620-631"},"PeriodicalIF":0.0,"publicationDate":"2024-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142650837","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-26DOI: 10.1016/j.mcpdig.2024.10.002
Gabrielle E. Koh MBBS , Brittany Ng MBBS , Ronja M.B. Lagström MSc , Fung-Joon Foo FRCS , Shuen-Ern Chin MBBS , Fang-Ting Wan MBBS , Juinn Huar Kam FRCS , Baldwin Yeung PhD, FRCS , Clarence Kwan MRCP , Cesare Hassan MD, PhD , Ismail Gögenur MD, DMSc , Frederick H. Koh FRCS, PhD
Objective
To review the efficacy and accuracy of the GI Genius Intelligent Endoscopy Module Computer-Assisted Diagnosis (CADx) program in colonic adenoma detection and real-time polyp characterization.
Patients and Methods
Colonoscopy remains the gold standard in colonic screening and evaluation. The incorporation of artificial intelligence (AI) technology therefore allows for optimized endoscopic performance. However, validation of most CADx programs with real-world data remains scarce. This prospective cohort study was conducted within a single Singaporean institution between April 1, 2023 and December 31, 2023. Videos of all AI-enabled colonoscopies were reviewed with polyp-by-polyp analysis performed. Real-time polyp characterization predictions after sustained polyp detection were compared against final histology results to assess the accuracy of the CADx system at colonic adenoma identification.
Results
A total of 808 videos of CADx colonoscopies were reviewed. Out of the 781 polypectomies performed, 543 (69.5%) and 222 (28.4%) were adenomas and non-adenomas on final histology, respectively. Overall, GI Genius correctly characterized adenomas with 89.4% sensitivity, 61.7% specificity, a positive predictive value of 85.4%, a negative predictive value of 69.8%, and 81.5% accuracy. The negative predictive value for rectosigmoid lesions (80.3%) was notably higher than for colonic lesions (54.2%), attributed to the increased prevalence of hyperplastic rectosigmoid polyps (11.4%) vs other colonic regions (5.4%).
Conclusion
Computer-Assisted Diagnosis is therefore a promising adjunct in colonoscopy with substantial clinical implications. Accurate identification of low-risk non-adenomatous polyps encourages the adoption of “resect-and-discard” strategies. However, further calibration of AI systems is needed before the acceptance of such strategies as the new standard of care.
{"title":"Real-World Assessment of the Efficacy of Computer-Assisted Diagnosis in Colonoscopy: A Single Institution Cohort Study in Singapore","authors":"Gabrielle E. Koh MBBS , Brittany Ng MBBS , Ronja M.B. Lagström MSc , Fung-Joon Foo FRCS , Shuen-Ern Chin MBBS , Fang-Ting Wan MBBS , Juinn Huar Kam FRCS , Baldwin Yeung PhD, FRCS , Clarence Kwan MRCP , Cesare Hassan MD, PhD , Ismail Gögenur MD, DMSc , Frederick H. Koh FRCS, PhD","doi":"10.1016/j.mcpdig.2024.10.002","DOIUrl":"10.1016/j.mcpdig.2024.10.002","url":null,"abstract":"<div><h3>Objective</h3><div>To review the efficacy and accuracy of the GI Genius Intelligent Endoscopy Module Computer-Assisted Diagnosis (CADx) program in colonic adenoma detection and real-time polyp characterization.</div></div><div><h3>Patients and Methods</h3><div>Colonoscopy remains the gold standard in colonic screening and evaluation. The incorporation of artificial intelligence (AI) technology therefore allows for optimized endoscopic performance. However, validation of most CADx programs with real-world data remains scarce. This prospective cohort study was conducted within a single Singaporean institution between April 1, 2023 and December 31, 2023. Videos of all AI-enabled colonoscopies were reviewed with polyp-by-polyp analysis performed. Real-time polyp characterization predictions after sustained polyp detection were compared against final histology results to assess the accuracy of the CADx system at colonic adenoma identification.</div></div><div><h3>Results</h3><div>A total of 808 videos of CADx colonoscopies were reviewed. Out of the 781 polypectomies performed, 543 (69.5%) and 222 (28.4%) were adenomas and non-adenomas on final histology, respectively. Overall, GI Genius correctly characterized adenomas with 89.4% sensitivity, 61.7% specificity, a positive predictive value of 85.4%, a negative predictive value of 69.8%, and 81.5% accuracy. The negative predictive value for rectosigmoid lesions (80.3%) was notably higher than for colonic lesions (54.2%), attributed to the increased prevalence of hyperplastic rectosigmoid polyps (11.4%) vs other colonic regions (5.4%).</div></div><div><h3>Conclusion</h3><div>Computer-Assisted Diagnosis is therefore a promising adjunct in colonoscopy with substantial clinical implications. Accurate identification of low-risk non-adenomatous polyps encourages the adoption of “resect-and-discard” strategies. However, further calibration of AI systems is needed before the acceptance of such strategies as the new standard of care.</div></div>","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 647-655"},"PeriodicalIF":0.0,"publicationDate":"2024-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142704499","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-19DOI: 10.1016/j.mcpdig.2024.09.006
Zilma Silveira Nogueira Reis MD, PhD , Adriana Silvina Pagano MA, PhD , Isaias Jose Ramos de Oliveira MSc , Cristiane dos Santos Dias MD, PhD , Eura Martins Lage MD, PhD , Erico Franco Mineiro PhD , Glaucia Miranda Varella Pereira PhD , Igor de Carvalho Gomes MSc , Vinicius Araujo Basilio MS , Ricardo João Cruz-Correia PhD , Davi dos Reis de Jesus BCS , Antônio Pereira de Souza Júnior MS , Leonardo Chaves Dutra da Rocha PhD
Objective
To assess the support of large language models (LLMs) in generating clearer and more personalized medication instructions to enhance e-prescription.
Patients and Methods
We established patient-centered guidelines for adequate, acceptable, and personalized directions to enhance e-prescription. A dataset comprising 104 outpatient scenarios, with an array of medications, administration routes, and patient conditions, was developed following the Brazilian national e-prescribing standard. Three prompts were submitted to a closed-source LLM. The first prompt involved a generic command, the second one was calibrated for content enhancement and personalization, and the third one requested bias mitigation. The third prompt was submitted to an open-source LLM. Outputs were assessed using automated metrics and human evaluation. We conducted the study between March 1, 2024 and September 10, 2024.
Results
Adequacy scores of our closed-source LLM’s output showed the third prompt outperforming the first and second one. Full and partial acceptability was achieved in 94.3% of texts with the third prompt. Personalization was rated highly, especially with the second and third prompts. The 2 LLMs showed similar adequacy results. Lack of scientific evidence and factual errors were infrequent and unrelated to a particular prompt or LLM. The frequency of hallucinations was different for each LLM and concerned prescriptions issued upon symptom manifestation and medications requiring dosage adjustment or involving intermittent use. Gender bias was found in our closed-source LLM’s output for the first and second prompts, with the third one being bias-free. The second LLM’s output was bias-free.
Conclusion
This study demonstrates the potential of LLM-supported generation to produce prescription directions and improve communication between health professionals and patients within the e-prescribing system.
{"title":"Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model","authors":"Zilma Silveira Nogueira Reis MD, PhD , Adriana Silvina Pagano MA, PhD , Isaias Jose Ramos de Oliveira MSc , Cristiane dos Santos Dias MD, PhD , Eura Martins Lage MD, PhD , Erico Franco Mineiro PhD , Glaucia Miranda Varella Pereira PhD , Igor de Carvalho Gomes MSc , Vinicius Araujo Basilio MS , Ricardo João Cruz-Correia PhD , Davi dos Reis de Jesus BCS , Antônio Pereira de Souza Júnior MS , Leonardo Chaves Dutra da Rocha PhD","doi":"10.1016/j.mcpdig.2024.09.006","DOIUrl":"10.1016/j.mcpdig.2024.09.006","url":null,"abstract":"<div><h3>Objective</h3><div>To assess the support of large language models (LLMs) in generating clearer and more personalized medication instructions to enhance e-prescription.</div></div><div><h3>Patients and Methods</h3><div>We established patient-centered guidelines for adequate, acceptable, and personalized directions to enhance e-prescription. A dataset comprising 104 outpatient scenarios, with an array of medications, administration routes, and patient conditions, was developed following the Brazilian national e-prescribing standard. Three prompts were submitted to a closed-source LLM. The first prompt involved a generic command, the second one was calibrated for content enhancement and personalization, and the third one requested bias mitigation. The third prompt was submitted to an open-source LLM. Outputs were assessed using automated metrics and human evaluation. We conducted the study between March 1, 2024 and September 10, 2024.</div></div><div><h3>Results</h3><div>Adequacy scores of our closed-source LLM’s output showed the third prompt outperforming the first and second one. Full and partial acceptability was achieved in 94.3% of texts with the third prompt. Personalization was rated highly, especially with the second and third prompts. The 2 LLMs showed similar adequacy results. Lack of scientific evidence and factual errors were infrequent and unrelated to a particular prompt or LLM. The frequency of hallucinations was different for each LLM and concerned prescriptions issued upon symptom manifestation and medications requiring dosage adjustment or involving intermittent use. Gender bias was found in our closed-source LLM’s output for the first and second prompts, with the third one being bias-free. The second LLM’s output was bias-free.</div></div><div><h3>Conclusion</h3><div>This study demonstrates the potential of LLM-supported generation to produce prescription directions and improve communication between health professionals and patients within the e-prescribing system.</div></div>","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 632-644"},"PeriodicalIF":0.0,"publicationDate":"2024-10-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142704498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-11DOI: 10.1016/j.mcpdig.2024.09.005
Md Saifur Rahman PhD , Chandan Karmarkar PhD , Sheikh Mohammed Shariful Islam MBBS, PhD
{"title":"Application of Federated Learning in Cardiology: Key Challenges and Potential Solutions","authors":"Md Saifur Rahman PhD , Chandan Karmarkar PhD , Sheikh Mohammed Shariful Islam MBBS, PhD","doi":"10.1016/j.mcpdig.2024.09.005","DOIUrl":"10.1016/j.mcpdig.2024.09.005","url":null,"abstract":"","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 590-595"},"PeriodicalIF":0.0,"publicationDate":"2024-10-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142552432","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-10DOI: 10.1016/j.mcpdig.2024.09.004
Donal J. Sexton MD, PhD , Conor Judge MB, PhD
{"title":"Assessments of Generative Artificial Intelligence as Clinical Decision Support Ought to be Incorporated Into Randomized Controlled Trials of Electronic Alerts for Acute Kidney Injury","authors":"Donal J. Sexton MD, PhD , Conor Judge MB, PhD","doi":"10.1016/j.mcpdig.2024.09.004","DOIUrl":"10.1016/j.mcpdig.2024.09.004","url":null,"abstract":"","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 606-610"},"PeriodicalIF":0.0,"publicationDate":"2024-10-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142592645","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-10-09DOI: 10.1016/j.mcpdig.2024.09.003
Tahani M. Ahmad MD, ABR , Alessandro Guida PhD , Sam Stewart PhD , Noah Barrett MSc , Michael J. Vincer MD , Jehier K. Afifi MD, MSc
Objective
To develop deep learning (DL) models applied to neonatal cranial ultrasound (CUS) and clinical variables to predict neurodevelopmental impairment (NDI) in very preterm infants (VPIs) at 3 years of corrected age.
Patients and Methods
This is a retrospective study of a cohort of VPI (220-306 weeks’ gestation) born between 2004 and 2016 in Nova Scotia, Canada. Clinical data at hospital discharge and CUS images at 3 time points were used to develop DL models using elastic net (EN) and convolutional neural network (CNN). The models’ performances were compared using precision recall area under the curve (PR-AUC) and area under the receiver operation characteristic curve (ROC-AUC) with their 95% ci.
Results
Of 665 eligible VPIs, 619 (93%) infants with 4184 CUS images were included. The CNN model combining CUS and clinical variables reported better performance (PR-AUC, 0.75; 95% CI, 072-0.79; ROC-AUC, 0.71; 95% CI, 0.67-0.74) in the prediction of positive NDI outcome compared with the traditional models based solely on clinical predictors (PR-AUC, 0.60; 95% CI, 0.52-0.68; ROC-AUC, 0.72; 95% CI, 0.68-0.75). When analyzed by the CUS plane and acquisition time point, the model using the anterior coronal plane at 6 weeks of age provided the highest predictive accuracy (PR-AUC, 0.81; 95% CI, 0.77-0.91; ROC-AUC, 0.78; 95% CI, 0.66-0.87).
Conclusion
We developed and internally validated a DL prognostic model using CUS and clinical predictors to predict NDI in VPIs at 3 years of age. Early and accurate identification of infants at risk for NDI enables referral to targeted interventions, which improves functional outcomes.
{"title":"Deep Learning Model for Predicting Neurodevelopmental Outcome in Very Preterm Infants Using Cerebral Ultrasound","authors":"Tahani M. Ahmad MD, ABR , Alessandro Guida PhD , Sam Stewart PhD , Noah Barrett MSc , Michael J. Vincer MD , Jehier K. Afifi MD, MSc","doi":"10.1016/j.mcpdig.2024.09.003","DOIUrl":"10.1016/j.mcpdig.2024.09.003","url":null,"abstract":"<div><h3>Objective</h3><div>To develop deep learning (DL) models applied to neonatal cranial ultrasound (CUS) and clinical variables to predict neurodevelopmental impairment (NDI) in very preterm infants (VPIs) at 3 years of corrected age.</div></div><div><h3>Patients and Methods</h3><div>This is a retrospective study of a cohort of VPI (22<sup>0</sup>-30<sup>6</sup> weeks’ gestation) born between 2004 and 2016 in Nova Scotia, Canada. Clinical data at hospital discharge and CUS images at 3 time points were used to develop DL models using elastic net (EN) and convolutional neural network (CNN). The models’ performances were compared using precision recall area under the curve (PR-AUC) and area under the receiver operation characteristic curve (ROC-AUC) with their 95% ci.</div></div><div><h3>Results</h3><div>Of 665 eligible VPIs, 619 (93%) infants with 4184 CUS images were included. The CNN model combining CUS and clinical variables reported better performance (PR-AUC, 0.75; 95% CI, 072-0.79; ROC-AUC, 0.71; 95% CI, 0.67-0.74) in the prediction of positive NDI outcome compared with the traditional models based solely on clinical predictors (PR-AUC, 0.60; 95% CI, 0.52-0.68; ROC-AUC, 0.72; 95% CI, 0.68-0.75). When analyzed by the CUS plane and acquisition time point, the model using the anterior coronal plane at 6 weeks of age provided the highest predictive accuracy (PR-AUC, 0.81; 95% CI, 0.77-0.91; ROC-AUC, 0.78; 95% CI, 0.66-0.87).</div></div><div><h3>Conclusion</h3><div>We developed and internally validated a DL prognostic model using CUS and clinical predictors to predict NDI in VPIs at 3 years of age. Early and accurate identification of infants at risk for NDI enables referral to targeted interventions, which improves functional outcomes.</div></div>","PeriodicalId":74127,"journal":{"name":"Mayo Clinic Proceedings. Digital health","volume":"2 4","pages":"Pages 596-605"},"PeriodicalIF":0.0,"publicationDate":"2024-10-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142552431","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}