Accessing information provided via artificial intelligence regarding reverse and anatomic total shoulder arthroplasty

Abstract

Background

The purpose of this study is to analyze the quality, accuracy, and readability of information provided by an artificial intelligence (AI) interface ChatGPT (OpenAI, San Francisco). We searched ChatGPT for commonly asked questions by patients regarding anatomic total shoulder arthroplasty (aTSA) and reverse total shoulder arthroplasty (rTSA).

Methods

ChatGPT was used to answer 30 commonly asked questions by patients regarding aTSA and rTSA, inputted as “total shoulder replacement” and “reverse shoulder replacement”. These questions were categorized based on the Rothwell criteria into Fact, Policy, and Value. The answers generated were analyzed for quality, accuracy, and readability using the DISCERN scale, JAMA benchmark criteria, Flesch-Kincaid Reading Ease Score (FRES) and Grade Level (FKGL).

Results

For both rTSA and aTSA the DISCERN score for Fact questions was 57, Policy questions was 61, and for Value questions was 58 (all were all considered “good”). The JAMA benchmark criteria was 0, representing the lowest score for Fact, Policy, and Value questions for both rTSA and aTSA questions. The FRES score for the aTSA answers for Fact was 15.15, for Policy was 11.14, and for Value questions was 10.95. The FRES score for rTSA questions for Fact is 48.02, Policy is 12.51, and Value is 17.22. The FKGL for aTSA answer for Fact was 17.48, Policy was 17.72 and Value was 17.96. The FKGL for rTSA questions for Fact are 8.10, Policy is 17.27, and Value is 16.56.

Conclusion

Overall, the quality of answers provided by AI open model, ChatGPT was considered “good.” The information provided had lower reliability, and lack of information regarding funding and disclosures. Most of the information generated by ChatGPT was also found to have the readability of “academic level text”, while Fact related information on reverse shoulder arthroplasty was found to have the readability of 9th grade level, which may be too complex for most patients. Overall, these results indicate that ChatGPT can provide correct answers to questions about aTSA and rTSA, although we would caution patients from utilizing this resource due to the lack of citations and complexity of the output that ChatGPT provides. Importantly, all answers provided by AI suggested reaching out to physicians to get more accurate and personalized advise, to factor into the shared decisions making model.