Cameron C. Young, Ellie Enichen, Christian Rivera, Corinne A. Auger, Nathan Grant, Arya Rao, Marc D. Succi
Accurately diagnosing rare pediatric diseases frequently represent a clinical challenge due to their complex and unusual clinical presentations. Here, we explore the capabilities of three large language models (LLMs), GPT‐4, Gemini Pro, and a custom‐built LLM (GPT‐4 integrated with the Human Phenotype Ontology [GPT‐4 HPO]), by evaluating their diagnostic performance on 61 rare pediatric disease case reports. The performance of the LLMs were assessed for accuracy in identifying specific diagnoses, listing the correct diagnosis among a differential list, and broad disease categories. In addition, GPT‐4 HPO was tested on 100 general pediatrics case reports previously assessed on other LLMs to further validate its performance. The results indicated that GPT‐4 was able to predict the correct diagnosis with a diagnostic accuracy of 13.1%, whereas both GPT‐4 HPO and Gemini Pro had diagnostic accuracies of 8.2%. Further, GPT‐4 HPO showed an improved performance compared with the other two LLMs in identifying the correct diagnosis among its differential list and the broad disease category. Although these findings underscore the potential of LLMs for diagnostic support, particularly when enhanced with domain‐specific ontologies, they also stress the need for further improvement prior to integration into clinical practice.
{"title":"Diagnostic Accuracy of a Custom Large Language Model on Rare Pediatric Disease Case Reports","authors":"Cameron C. Young, Ellie Enichen, Christian Rivera, Corinne A. Auger, Nathan Grant, Arya Rao, Marc D. Succi","doi":"10.1002/ajmg.a.63878","DOIUrl":"https://doi.org/10.1002/ajmg.a.63878","url":null,"abstract":"Accurately diagnosing rare pediatric diseases frequently represent a clinical challenge due to their complex and unusual clinical presentations. Here, we explore the capabilities of three large language models (LLMs), GPT‐4, Gemini Pro, and a custom‐built LLM (GPT‐4 integrated with the Human Phenotype Ontology [GPT‐4 HPO]), by evaluating their diagnostic performance on 61 rare pediatric disease case reports. The performance of the LLMs were assessed for accuracy in identifying specific diagnoses, listing the correct diagnosis among a differential list, and broad disease categories. In addition, GPT‐4 HPO was tested on 100 general pediatrics case reports previously assessed on other LLMs to further validate its performance. The results indicated that GPT‐4 was able to predict the correct diagnosis with a diagnostic accuracy of 13.1%, whereas both GPT‐4 HPO and Gemini Pro had diagnostic accuracies of 8.2%. Further, GPT‐4 HPO showed an improved performance compared with the other two LLMs in identifying the correct diagnosis among its differential list and the broad disease category. Although these findings underscore the potential of LLMs for diagnostic support, particularly when enhanced with domain‐specific ontologies, they also stress the need for further improvement prior to integration into clinical practice.","PeriodicalId":7507,"journal":{"name":"American Journal of Medical Genetics Part A","volume":null,"pages":null},"PeriodicalIF":2.0,"publicationDate":"2024-09-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142251255","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Study Strengthens Link between Autism Spectrum Disorder and Gut Microbiome","authors":"","doi":"10.1002/ajmg.a.63288","DOIUrl":"https://doi.org/10.1002/ajmg.a.63288","url":null,"abstract":"","PeriodicalId":7507,"journal":{"name":"American Journal of Medical Genetics Part A","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142174211","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Table of Contents, Volume 194A, Number 10, October 2024","authors":"","doi":"10.1002/ajmg.a.63291","DOIUrl":"https://doi.org/10.1002/ajmg.a.63291","url":null,"abstract":"","PeriodicalId":7507,"journal":{"name":"American Journal of Medical Genetics Part A","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/ajmg.a.63291","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142174155","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Whole-Genome Sequencing Can Improve Care in Pediatric Cancer","authors":"","doi":"10.1002/ajmg.a.63289","DOIUrl":"https://doi.org/10.1002/ajmg.a.63289","url":null,"abstract":"","PeriodicalId":7507,"journal":{"name":"American Journal of Medical Genetics Part A","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142174229","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The cover image is based on the article Germline PTCH1: c.361_362insAlu alteration identified by comprehensive exome and RNA sequencing in a patient with Gorlin syndrome by Aaron Y. Mochizuki et al., https://doi.org/10.1002/ajmg.a.63788.