Michael Megafu, Omar Guerrero, Avanish Yendluri, Bradford O Parsons, Leesa M Galatz, Xinning Li, John D Kelly, Robert L Parisien
{"title":"ChatGPT and Gemini Are Not Consistently Concordant with 2020 AAOS Clinical Practice Guidelines when Evaluating Rotator Cuff Injury.","authors":"Michael Megafu, Omar Guerrero, Avanish Yendluri, Bradford O Parsons, Leesa M Galatz, Xinning Li, John D Kelly, Robert L Parisien","doi":"10.1016/j.arthro.2025.01.039","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>The objective of this study was to evaluate the accuracy of suggestions given by ChatGPT, developed by Open AI and Gemini (previously known as Bard), the two widely used publicly available language learning models (LLMs), to evaluate the management of rotator cuff injuries.</p><p><strong>Methods: </strong>The 2020 AAOS Clinical Practice Guidelines (CPG) were the basis for determining recommended and non-recommended treatments in this study. ChatGPT and Gemini were queried on 16 treatments based on these guidelines examining rotator cuff interventions. These responses were categorized as \"Concordant\" or \"Discordant\" with the AAOS CPG. A Cohen's Kappa coefficient was calculated to assess the interrater reliability.</p><p><strong>Results: </strong>Of the 16 treatments queried, ChatGPT and Gemini demonstrated concordance with the AAOS CPG for 13 (81%) and 12 (75%) treatments, respectively. ChatGPT provided discordant responses with the AAOS CPG for three treatments (19%), while Gemini provided discordant responses for four treatments (25%). Using Cohen's Kappa coefficient, the interrater reliability was 0.98 between the raters, signifying agreement in classifying the responses of ChatGPT and Gemini to the AAOS CPG as being \"concordant\" or \"discordant.\"</p><p><strong>Conclusion: </strong>ChatGPT and Gemini do not consistently provide responses that align with the AAOS CPG.</p>","PeriodicalId":55459,"journal":{"name":"Arthroscopy-The Journal of Arthroscopic and Related Surgery","volume":" ","pages":""},"PeriodicalIF":4.4000,"publicationDate":"2025-02-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Arthroscopy-The Journal of Arthroscopic and Related Surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.arthro.2025.01.039","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ORTHOPEDICS","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose: The objective of this study was to evaluate the accuracy of suggestions given by ChatGPT, developed by Open AI and Gemini (previously known as Bard), the two widely used publicly available language learning models (LLMs), to evaluate the management of rotator cuff injuries.
Methods: The 2020 AAOS Clinical Practice Guidelines (CPG) were the basis for determining recommended and non-recommended treatments in this study. ChatGPT and Gemini were queried on 16 treatments based on these guidelines examining rotator cuff interventions. These responses were categorized as "Concordant" or "Discordant" with the AAOS CPG. A Cohen's Kappa coefficient was calculated to assess the interrater reliability.
Results: Of the 16 treatments queried, ChatGPT and Gemini demonstrated concordance with the AAOS CPG for 13 (81%) and 12 (75%) treatments, respectively. ChatGPT provided discordant responses with the AAOS CPG for three treatments (19%), while Gemini provided discordant responses for four treatments (25%). Using Cohen's Kappa coefficient, the interrater reliability was 0.98 between the raters, signifying agreement in classifying the responses of ChatGPT and Gemini to the AAOS CPG as being "concordant" or "discordant."
Conclusion: ChatGPT and Gemini do not consistently provide responses that align with the AAOS CPG.
期刊介绍：
Nowhere is minimally invasive surgery explained better than in Arthroscopy, the leading peer-reviewed journal in the field. Every issue enables you to put into perspective the usefulness of the various emerging arthroscopic techniques. The advantages and disadvantages of these methods -- along with their applications in various situations -- are discussed in relation to their efficiency, efficacy and cost benefit. As a special incentive, paid subscribers also receive access to the journal expanded website.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
