ChatGPT and Gemini Are Not Consistently Concordant With the 2020 American Academy of Orthopaedic Surgeons Clinical Practice Guidelines When Evaluating Rotator Cuff Injury

Abstract

Purpose

To evaluate the accuracy of suggestions given by ChatGPT and Gemini (previously known as “Bard”), 2 widely used publicly available large language models, to evaluate the management of rotator cuff injuries.

Methods

The 2020 American Academy of Orthopaedic Surgeons (AAOS) Clinical Practice Guidelines (CPGs) were the basis for determining recommended and non-recommended treatments in this study. ChatGPT and Gemini were queried on 16 treatments based on these guidelines examining rotator cuff interventions. The responses were categorized as “concordant” or “discordant” with the AAOS CPGs. The Cohen κ coefficient was calculated to assess inter-rater reliability.

Results

ChatGPT and Gemini showed concordance with the AAOS CPGs for 13 of the 16 treatments queried (81%) and 12 of the 16 treatments queried (75%), respectively. ChatGPT provided discordant responses with the AAOS CPGs for 3 treatments (19%), whereas Gemini provided discordant responses for 4 treatments (25%). Assessment of inter-rater reliability showed a Cohen κ coefficient of 0.98, signifying agreement between the raters in classifying the responses of ChatGPT and Gemini to the AAOS CPGs as being concordant or discordant.

Conclusions

ChatGPT and Gemini do not consistently provide responses that align with the AAOS CPGs.

Clinical Relevance

This study provides evidence that cautions patients not to rely solely on artificial intelligence for recommendations about rotator cuff injuries.