Artificial Intelligence Shows Limited Success in Improving Readability Levels of Spanish-language Orthopaedic Patient Education Materials.

Abstract

Background: The more than 41 million people in the United States who speak Spanish represent one of the fastest-growing US populations. Non-English-speaking patients often face poorer health outcomes because of language barriers that hinder patient education. Orthopaedic education materials have limited availability in Spanish and may be difficult for some patients to read. The American Academy of Orthopaedic Surgeons (AAOS) has translated education materials into Spanish, but their readability levels remain unknown. Additionally, although artificial intelligence (AI) dialogue platforms have been shown to improve readability in English, no studies have specifically evaluated their effectiveness in non-English languages.

Questions/purposes: (1) What is the readability of AAOS Spanish-language education materials? (2) Can an AI dialogue platform improve the readability of Spanish-language education materials while maintaining their accuracy and usefulness?

Methods: After excluding COVID-19 articles and inaccessible websites, Spanish-language education materials were extracted from the AAOS OrthoInfo website, and their Fernández-Huerta and Spanish Orthographic Length (SOL) readability grade levels were calculated. Fernández-Huerta focuses on syntactic complexity (sentence and syllable structure) and SOL assesses lexical complexity (word length and frequency). For both, the higher the grade level, the harder it is to read. Education materials with a reading level above the sixth-grade level were inputted into the ChatGPT-4 AI platform to be adapted to a fifth-grade level. Readability metrics of the adaptations were reassessed and compared with the original versions. Secondarily, one of four Spanish-speaking orthopaedic surgeons evaluated each AI-adapted education material for accuracy and usefulness compared with the original version. We used a single review per material, trusting the orthopaedic surgeon's expertise to minimize discrepancies. We included a total of 77 of 82 education materials covering topics like diseases and conditions, treatment, and recovery and staying healthy.

Results: Before AI adaptations, none of the 77 education materials met the recommended reading level of sixth grade or below according to both readability formulas. The original education materials were written at a seventh- to eighth-grade reading level in 32% of cases (25 of 77). In comparison, after a single attempt at simplification, AI-adapted materials achieved this reading level in 53% of cases (41 of 77; p < 0.001). Only 23% (18) and 16% (12) of the AI adaptations were written at or below the recommended sixth-grade level per the Fernández-Huerta and SOL grade levels, respectively. Of the AI adaptations, 52% (40) were rated as accurate and 56% (43) were rated as useful for patient education by the evaluating orthopaedic surgeons. AI adaptations that were classified as accurate or useful had a higher median (IQR) word count than those that were inaccurate (accurate 255 [216 to 331] versus inaccurate 236 [209 to 256]; p = 0.04) or not useful (useful 257 [216 to 337] versus not useful 233 [209 to 251]; p = 0.01).

Conclusion: Ongoing attention is needed to improve the readability of Spanish education materials to reduce health disparities. ChatGPT-4 has limited success in improving readability without compromising accuracy and usefulness. We urge AAOS to enhance the readability of these materials and recommend physicians use them as supplemental resources while prioritizing direct patient education for Spanish-speaking individuals. Further research is needed to develop readable and culturally appropriate education materials for non-English-speaking patients that incorporate direct patient feedback.

Clinical relevance: This study shows that Spanish-language orthopaedic materials often exceed recommended readability levels, limiting their effectiveness and worsening health disparities. While AI tools like ChatGPT-4 improve readability, they may fall short in accuracy and usefulness. This underscores the need for clearer, culturally appropriate materials and the importance of physicians providing direct education.