High identification and positive-negative discrimination but limited detailed grading accuracy of ChatGPT-4o in knee osteoarthritis radiographs.

IF 3.3 2区医学 Q1 ORTHOPEDICS Knee Surgery, Sports Traumatology, Arthroscopy Pub Date : 2025-03-07 DOI:10.1002/ksa.12639

Jiesheng Zhu, Yilun Jiang, Daosen Chen, Yi Lu, Yijiang Huang, Yimu Lin, Pei Fan

{"title":"High identification and positive-negative discrimination but limited detailed grading accuracy of ChatGPT-4o in knee osteoarthritis radiographs.","authors":"Jiesheng Zhu, Yilun Jiang, Daosen Chen, Yi Lu, Yijiang Huang, Yimu Lin, Pei Fan","doi":"10.1002/ksa.12639","DOIUrl":null,"url":null,"abstract":"Purpose: To explore the potential of ChatGPT-4o in analysing radiographic images of knee osteoarthritis (OA) and to assess its grading accuracy, feature identification and reliability, thereby helping surgeons to improve diagnostic accuracy and efficiency.Methods: A total of 117 anterior‒posterior knee radiographs from patients (23.1% men, 76.9% women, mean age 69.7 ± 7.99 years) were analysed. Two senior orthopaedic surgeons and ChatGPT-4o independently graded images with the Kellgren-Lawrence (K-L), Ahlbäck and International Knee Documentation Committee (IKDC) systems. A consensus reference standard was established by a third radiologist. ChatGPT-4o's performance metrics (accuracy, precision, recall and F1 score) were calculated, and its reliability was assessed via two evaluations separated by a 2-week interval, with intraclass correlation coefficients (ICCs) determined.Results: ChatGPT-4o achieved a 100% identification rate for knee radiographs and demonstrated strong binary classification performance (precision: 0.95, recall: 0.83, F score: 0.88). However, its detailed grading accuracy (35%) was substantially lower than that of surgeons (89.6%). Severe underestimation of OA severity occurred in 49.3% of the cases. Interrater reliability for surgeons was excellent (ICC: 0.78-0.91), whereas ChatGPT-4o showed poor initial consistency (ICC: 0.16-0.28), improving marginally in the second evaluation (ICC: 0.22-0.39).Conclusion: ChatGPT-4o has the potential to rapidly identify and binary classify knee OA on radiographs. However, its detailed grading accuracy remains suboptimal, with a notable tendency to underestimate severe cases. This limits its current clinical utility for precise staging. Future research should focus on optimising its grading performance and improving accuracy to enhance diagnostic reliability.Level of evidence: Level III retrospective comparative study.","PeriodicalId":17880,"journal":{"name":"Knee Surgery, Sports Traumatology, Arthroscopy","volume":" ","pages":""},"PeriodicalIF":3.3000,"publicationDate":"2025-03-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Knee Surgery, Sports Traumatology, Arthroscopy","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1002/ksa.12639","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ORTHOPEDICS","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: To explore the potential of ChatGPT-4o in analysing radiographic images of knee osteoarthritis (OA) and to assess its grading accuracy, feature identification and reliability, thereby helping surgeons to improve diagnostic accuracy and efficiency.

Methods: A total of 117 anterior‒posterior knee radiographs from patients (23.1% men, 76.9% women, mean age 69.7 ± 7.99 years) were analysed. Two senior orthopaedic surgeons and ChatGPT-4o independently graded images with the Kellgren-Lawrence (K-L), Ahlbäck and International Knee Documentation Committee (IKDC) systems. A consensus reference standard was established by a third radiologist. ChatGPT-4o's performance metrics (accuracy, precision, recall and F1 score) were calculated, and its reliability was assessed via two evaluations separated by a 2-week interval, with intraclass correlation coefficients (ICCs) determined.

Results: ChatGPT-4o achieved a 100% identification rate for knee radiographs and demonstrated strong binary classification performance (precision: 0.95, recall: 0.83, F score: 0.88). However, its detailed grading accuracy (35%) was substantially lower than that of surgeons (89.6%). Severe underestimation of OA severity occurred in 49.3% of the cases. Interrater reliability for surgeons was excellent (ICC: 0.78-0.91), whereas ChatGPT-4o showed poor initial consistency (ICC: 0.16-0.28), improving marginally in the second evaluation (ICC: 0.22-0.39).

Conclusion: ChatGPT-4o has the potential to rapidly identify and binary classify knee OA on radiographs. However, its detailed grading accuracy remains suboptimal, with a notable tendency to underestimate severe cases. This limits its current clinical utility for precise staging. Future research should focus on optimising its grading performance and improving accuracy to enhance diagnostic reliability.

Level of evidence: Level III retrospective comparative study.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

求助全文

约1分钟内获得全文去求助

来源期刊

Knee Surgery, Sports Traumatology, Arthroscopy 医学-外科

CiteScore

8.10

自引率

18.40%

发文量

418

审稿时长

2 months

期刊介绍： Few other areas of orthopedic surgery and traumatology have undergone such a dramatic evolution in the last 10 years as knee surgery, arthroscopy and sports traumatology. Ranked among the top 33% of journals in both Orthopedics and Sports Sciences, the goal of this European journal is to publish papers about innovative knee surgery, sports trauma surgery and arthroscopy. Each issue features a series of peer-reviewed articles that deal with diagnosis and management and with basic research. Each issue also contains at least one review article about an important clinical problem. Case presentations or short notes about technical innovations are also accepted for publication. The articles cover all aspects of knee surgery and all types of sports trauma; in addition, epidemiology, diagnosis, treatment and prevention, and all types of arthroscopy (not only the knee but also the shoulder, elbow, wrist, hip, ankle, etc.) are addressed. Articles on new diagnostic techniques such as MRI and ultrasound and high-quality articles about the biomechanics of joints, muscles and tendons are included. Although this is largely a clinical journal, it is also open to basic research with clinical relevance. Because the journal is supported by a distinguished European Editorial Board, assisted by an international Advisory Board, you can be assured that the journal maintains the highest standards. Official Clinical Journal of the European Society of Sports Traumatology, Knee Surgery and Arthroscopy (ESSKA).