Assessment and comparison of artificial intelligence–generated information regarding shoulder arthroplasty from multiple interfaces

Abstract

Background

This study aims to analyze and compare the quality, accuracy, and readability of information regarding anatomic total shoulder arthroplasty (aTSA) and reverse total shoulder arthroplasty (rTSA) provided by various AI interfaces (Open AI’s ChatGPT and Microsoft’s CoPilot).

Methods

Thirty commonly asked questions (categorized by Rothwell criteria into Fact, Policy, and Value) by patients were inputted into ChatGPT 3.5 and CoPilot. Responses were assessed with the DISCERN scale, Journal of the American Medical Association (JAMA) benchmark criteria, and Flesch-Kincaid Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). The sources of citations provided by CoPilot were further analyzed.

Results

Both AI interfaces generated DISCERN scores >50 (aTSA and rTSA ChatGPT: 57 [Fact], 61 [Policy], 58 [Value]; aTSA and rTSA CoPilot: 68 [Fact], 72 [Policy], 70 [Value]), demonstrating “good” quality of information provided, except for the Policy questions by CoPilot, which were scored as “excellent” (>70). CoPilot's higher JAMA score (3 vs. 0) and FRES scores >30 indicated more reliable, accessible responses, which required a minimum of 12th-grade education to read the same. In comparison, the ChatGPT generated more complex texts, with the majority of the FRES scores <20, and FKGL score signifying complexity of academic level text. Finally, CoPilot provided citations and demonstrated the highest percentage of academic sources (31.1% for rTSA and 26.7% for aTSA), suggesting reliable sources of information.

Conclusion

Overall, the information provided by both AI interfaces ChatGPT and CoPilot was scored as a “good” source of information for commonly asked patient questions regarding shoulder arthroplasty. But the answers to questions pertaining to shoulder arthroplasty provided by CoPilot proved to be more reliable (P = .0061), less complex, easier to read (P = .0031), and referenced information from reliable resources including academic sources, journal articles, and medical sites. Although answers provided by CoPilot were “easier” to read, they still required a 12th-grade education, which may be too complex for most patients, posing a challenge for patient comprehension. There were a substantial amount of nonmedical media sites, and commercial sources that were cited for both aTSA and rTSA questions by CoPilot. Critically, answers from both AI interfaces should serve as supplementary resources rather than primary sources on perioperative conditions pertaining to shoulder arthroplasty.