Nur Dokuzeylul Gungor, Fatih Sinan Esen, Tolga Tasci, Kagan Gungor, Kaan Cil
{"title":"Navigating Gynecological Oncology with Different Versions of ChatGPT: A Transformative Breakthrough or the Next Black Box Challenge?","authors":"Nur Dokuzeylul Gungor, Fatih Sinan Esen, Tolga Tasci, Kagan Gungor, Kaan Cil","doi":"10.1159/000543173","DOIUrl":null,"url":null,"abstract":"<p><strong>Introduction: </strong>The study evaluates the performance of large language model versions of ChatGPT - ChatGPT-3.5, ChatGPT-4, and ChatGPT-Omni - in addressing inquiries related to the diagnosis and treatment of gynecological cancers, including ovarian, endometrial, and cervical cancers.</p><p><strong>Methods: </strong>A total of 804 questions were equally distributed across four categories: true/false, multiple-choice, open-ended, and case-scenario, with each question type representing varying levels of complexity. Performance was assessed using a six-point Likert scale, focusing on accuracy, completeness, and alignment with established clinical guidelines.</p><p><strong>Results: </strong>For true/false queries, ChatGPT-Omni achieved accuracy rates of 100% for easy, 98% for medium, and 97% for complicated questions, higher than ChatGPT-4 (94%, 90%, 85%) and ChatGPT-3.5 (90%, 85%, 80%) (p = 0.041, 0.023, 0.014, respectively). In multiple-choice, ChatGPT-Omni maintained superior accuracy with 100% for easy, 98% for medium, and 93% for complicated queries, compared to ChatGPT-4 (92%, 88%, 80%) and ChatGPT-3.5 (85%, 80%, 70%) (p = 0.035, 0.028, 0.011). For open-ended questions, ChatGPT-Omni had mean Likert scores of 5.8 for easy, 5.5 for medium, and 5.2 for complex levels, outperforming ChatGPT-4 (5.4, 5.0, 4.5) and ChatGPT-3.5 (5.0, 4.5, 4.0) (p = 0.037, 0.026, 0.015). Similar trends were observed in case-scenario questions, where ChatGPT-Omni achieved scores of 5.6, 5.3, and 4.9 for easy, medium, and hard levels, respectively (p = 0.017, 0.008, 0.012).</p><p><strong>Conclusions: </strong>ChatGPT-Omni exhibited superior performance in responding to clinical queries related to gynecological cancers, underscoring its potential utility as a decision support tool and an educational resource in clinical practice.</p>","PeriodicalId":19543,"journal":{"name":"Oncology Research and Treatment","volume":" ","pages":"1-9"},"PeriodicalIF":2.0000,"publicationDate":"2024-12-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Oncology Research and Treatment","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1159/000543173","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ONCOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Introduction: The study evaluates the performance of large language model versions of ChatGPT - ChatGPT-3.5, ChatGPT-4, and ChatGPT-Omni - in addressing inquiries related to the diagnosis and treatment of gynecological cancers, including ovarian, endometrial, and cervical cancers.
Methods: A total of 804 questions were equally distributed across four categories: true/false, multiple-choice, open-ended, and case-scenario, with each question type representing varying levels of complexity. Performance was assessed using a six-point Likert scale, focusing on accuracy, completeness, and alignment with established clinical guidelines.
Results: For true/false queries, ChatGPT-Omni achieved accuracy rates of 100% for easy, 98% for medium, and 97% for complicated questions, higher than ChatGPT-4 (94%, 90%, 85%) and ChatGPT-3.5 (90%, 85%, 80%) (p = 0.041, 0.023, 0.014, respectively). In multiple-choice, ChatGPT-Omni maintained superior accuracy with 100% for easy, 98% for medium, and 93% for complicated queries, compared to ChatGPT-4 (92%, 88%, 80%) and ChatGPT-3.5 (85%, 80%, 70%) (p = 0.035, 0.028, 0.011). For open-ended questions, ChatGPT-Omni had mean Likert scores of 5.8 for easy, 5.5 for medium, and 5.2 for complex levels, outperforming ChatGPT-4 (5.4, 5.0, 4.5) and ChatGPT-3.5 (5.0, 4.5, 4.0) (p = 0.037, 0.026, 0.015). Similar trends were observed in case-scenario questions, where ChatGPT-Omni achieved scores of 5.6, 5.3, and 4.9 for easy, medium, and hard levels, respectively (p = 0.017, 0.008, 0.012).
Conclusions: ChatGPT-Omni exhibited superior performance in responding to clinical queries related to gynecological cancers, underscoring its potential utility as a decision support tool and an educational resource in clinical practice.
期刊介绍：
With the first issue in 2014, the journal ''Onkologie'' has changed its title to ''Oncology Research and Treatment''. By this change, publisher and editor set the scene for the further development of this interdisciplinary journal. The English title makes it clear that the articles are published in English – a logical step for the journal, which is listed in all relevant international databases. For excellent manuscripts, a ''Fast Track'' was introduced: The review is carried out within 2 weeks; after acceptance the papers are published online within 14 days and immediately released as ''Editor’s Choice'' to provide the authors with maximum visibility of their results. Interesting case reports are published in the section ''Novel Insights from Clinical Practice'' which clearly highlights the scientific advances which the report presents.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
