Evaluation of Chat Generative Pre-trained Transformer and Microsoft Copilot Performance on the American Society of Surgery of the Hand Self-Assessment Examinations

Abstract

Purpose

Artificial intelligence advancements have the potential to transform medical education and patient care. The increasing popularity of large language models has raised important questions regarding their accuracy and agreement with human users. The purpose of this study was to evaluate the performance of Chat Generative Pre-Trained Transformer (ChatGPT), versions 3.5 and 4, as well as Microsoft Copilot, which is powered by ChatGPT-4, on self-assessment examination questions for hand surgery and compare results between versions.

Methods

Input included 1,000 questions across 5 years (2015–2019) of self-assessment examinations provided by the American Society of Surgery of the Hand. The primary outcomes included correctness, the percentage concordance relative to other users, and whether an additional prompt was required. Secondary outcomes included accuracy according to question type and difficulty.

Results

All question formats including image-based questions were used for the analysis. ChatGPT-3.5 correctly answered 51.6% and ChatGPT-4 correctly answered 63.4%, which was a statistically significant difference. Microsoft Copilot correctly answered 59.9% and outperformed ChatGPT-3.5 but scored significantly lower than ChatGPT-4. However, ChatGPT-3.5 sided with an average of 72.2% users when correct and 62.1% when incorrect, compared to an average of 67.0% and 53.2% users, respectively, for ChatGPT-4. Microsoft Copilot sided with an average of 79.7% users when correct and 52.1% when incorrect. The highest scoring subject was Miscellaneous, and the lowest scoring subject was Neuromuscular in all versions.

Conclusions

In this study, ChatGPT-4 and Microsoft Copilot perform better on the hand surgery subspecialty examinations than did ChatGPT-3.5. Microsoft Copilot was more accurate than ChatGPT3.5 but less accurate than ChatGPT4. The ChatGPT-4 and Microsoft Copilot were able to “pass” the 2015–2019 American Society for Surgery of the Hand self-assessment examinations.

Clinical Relevance

While holding promise within medical education, caution should be used with large language models as more detailed evaluation of consistency is needed. Future studies should explore how these models perform across multiple trials and contexts to truly assess their reliability.