ChatGPT-3.5 and -4.0 Do Not Reliably Create Readable Patient Education Materials for Common Orthopaedic Upper- and Lower-Extremity Conditions

Abstract

Purpose

To investigate whether ChatGPT-3.5 and -4.0 can serve as a viable tool to create readable patient education materials for patients with common orthopaedic upper- and lower-extremity conditions.

Methods

Using ChatGPT versions 3.5 and 4.0, we asked the artificial intelligence program a series of 2 questions pertaining to patient education for 50 common orthopaedic upper-extremity pathologies and 50 common orthopaedic lower-extremity pathologies. Two templated questions were created and used for all conditions. Readability scores were calculated using the Python library Textstat. Multiple readability test scores were generated, and a consensus reading level was created taking into account the results of 8 reading tests.

Results

ChatGPT-3.5 produced only 2% and 4% of responses at the appropriate reading level for upper- and lower-extremity conditions, respectively, compared with 54% produced by ChatGPT-4.0 for both upper- and lower-extremity conditions (both P < .0001). After a priming phase, ChatGPT-3.5 did not produce any viable responses for either the upper- or lower-extremity conditions, compared with 64% for both upper- and lower-extremity conditions by ChatGPT-4.0 (both P < .0001). Additionally, ChatGPT-4.0 was more successful than ChatGPT-3.5 in producing viable responses both before and after a priming phase based on all available metrics for reading level (all P < .001), including the Automated Readability index, Coleman-Liau index, Dale-Chall formula, Flesch-Kincaid grade, Flesch Reading Ease score, Gunning Fog score, Linsear Write Formula score, and Simple Measure of Gobbledygook index.

Conclusions

Our results indicate that ChatGPT-3.5 and -4.0 unreliably created readable patient education materials for common orthopaedic upper- and lower-extremity conditions at the time of the study.

Clinical Relevance

The findings of this study suggest that ChatGPT, while constantly improving as evidenced by the advances from version 3.5 to version 4.0, should not be substituted for traditional methods of patient education at this time and, in its current state, may be used as a supplemental resource at the discretion of providers.