Can ChatGPT Reliably Answer the Most Common Patient Questions Regarding Total Shoulder Arthroplasty?

Abstract

Background: Increasingly, patients are turning to artificial intelligence (AI) programs such as ChatGPT to answer medical questions either before or after consulting a physician. Although ChatGPT's popularity implies its potential in improving patient education, concerns exist regarding the validity of the chatbot's responses. Therefore, the objective of this study was to evaluate the quality and accuracy of ChatGPT's answers to commonly asked patient questions surrounding total shoulder arthroplasty (TSA).

Methods: Eleven trusted healthcare websites were searched to compose a list of the 15 most frequently asked patient questions about TSA. Each question was posed to the ChatGPT user interface, with no follow-up questions or opportunity for clarification permitted. Individual response accuracy was graded by three board-certified orthopedic surgeons using an alphabetical grading system (i.e., A-F). Overall grades, descriptive analyses, and commentary were provided for each of the ChatGPT responses.

Results: Overall, ChatGPT received a cumulative grade of B-. The question responses surrounding general/preoperative and postoperative questions received a grade of B- and B-, respectively. ChatGPT's responses adequately responded to patient questions with sound recommendations. However, the chatbot neglected recent research in its responses, resulting in recommendations that warrant professional clarification. The interface deferred specific questions to orthopedic surgeons in 8/15 questions, suggesting its awareness of its own limitations. Moreover, ChatGPT often went beyond the scope of the question after the first two sentences, and generally made errors when attempting to supplement its own response.

Conclusion: Overall, this is the first study to our knowledge to utilize AI to answer the most common patient questions surrounding TSA. ChatGPT achieved an overall grade of B-. Ultimately, while AI is an attractive tool for initial patient inquiries, at this time it cannot provide responses to TSA-specific questions that can substitute the knowledge of an orthopedic surgeon.