Enhancing Health Literacy: Evaluating the Readability of Patient Handouts Revised by ChatGPT's Large Language Model.

IF 2.6 3区 医学 Q1 OTORHINOLARYNGOLOGY Otolaryngology- Head and Neck Surgery Pub Date : 2024-12-01 Epub Date: 2024-08-06 DOI:10.1002/ohn.927
Austin R Swisher, Arthur W Wu, Gene C Liu, Matthew K Lee, Taylor R Carle, Dennis M Tang
{"title":"Enhancing Health Literacy: Evaluating the Readability of Patient Handouts Revised by ChatGPT's Large Language Model.","authors":"Austin R Swisher, Arthur W Wu, Gene C Liu, Matthew K Lee, Taylor R Carle, Dennis M Tang","doi":"10.1002/ohn.927","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>To use an artificial intelligence (AI)-powered large language model (LLM) to improve readability of patient handouts.</p><p><strong>Study design: </strong>Review of online material modified by AI.</p><p><strong>Setting: </strong>Academic center.</p><p><strong>Methods: </strong>Five handout materials obtained from the American Rhinologic Society (ARS) and the American Academy of Facial Plastic and Reconstructive Surgery websites were assessed using validated readability metrics. The handouts were inputted into OpenAI's ChatGPT-4 after prompting: \"Rewrite the following at a 6th-grade reading level.\" The understandability and actionability of both native and LLM-revised versions were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Results were compared using Wilcoxon rank-sum tests.</p><p><strong>Results: </strong>The mean readability scores of the standard (ARS, American Academy of Facial Plastic and Reconstructive Surgery) materials corresponded to \"difficult,\" with reading categories ranging between high school and university grade levels. Conversely, the LLM-revised handouts had an average seventh-grade reading level. LLM-revised handouts had better readability in nearly all metrics tested: Flesch-Kincaid Reading Ease (70.8 vs 43.9; P < .05), Gunning Fog Score (10.2 vs 14.42; P < .05), Simple Measure of Gobbledygook (9.9 vs 13.1; P < .05), Coleman-Liau (8.8 vs 12.6; P < .05), and Automated Readability Index (8.2 vs 10.7; P = .06). PEMAT scores were significantly higher in the LLM-revised handouts for understandability (91 vs 74%; P < .05) with similar actionability (42 vs 34%; P = .15) when compared to the standard materials.</p><p><strong>Conclusion: </strong>Patient-facing handouts can be augmented by ChatGPT with simple prompting to tailor information with improved readability. This study demonstrates the utility of LLMs to aid in rewriting patient handouts and may serve as a tool to help optimize education materials.</p><p><strong>Level of evidence: </strong>Level VI.</p>","PeriodicalId":19707,"journal":{"name":"Otolaryngology- Head and Neck Surgery","volume":" ","pages":"1751-1757"},"PeriodicalIF":2.6000,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Otolaryngology- Head and Neck Surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1002/ohn.927","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/8/6 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"OTORHINOLARYNGOLOGY","Score":null,"Total":0}
引用次数: 0

Abstract

Objective: To use an artificial intelligence (AI)-powered large language model (LLM) to improve readability of patient handouts.

Study design: Review of online material modified by AI.

Setting: Academic center.

Methods: Five handout materials obtained from the American Rhinologic Society (ARS) and the American Academy of Facial Plastic and Reconstructive Surgery websites were assessed using validated readability metrics. The handouts were inputted into OpenAI's ChatGPT-4 after prompting: "Rewrite the following at a 6th-grade reading level." The understandability and actionability of both native and LLM-revised versions were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Results were compared using Wilcoxon rank-sum tests.

Results: The mean readability scores of the standard (ARS, American Academy of Facial Plastic and Reconstructive Surgery) materials corresponded to "difficult," with reading categories ranging between high school and university grade levels. Conversely, the LLM-revised handouts had an average seventh-grade reading level. LLM-revised handouts had better readability in nearly all metrics tested: Flesch-Kincaid Reading Ease (70.8 vs 43.9; P < .05), Gunning Fog Score (10.2 vs 14.42; P < .05), Simple Measure of Gobbledygook (9.9 vs 13.1; P < .05), Coleman-Liau (8.8 vs 12.6; P < .05), and Automated Readability Index (8.2 vs 10.7; P = .06). PEMAT scores were significantly higher in the LLM-revised handouts for understandability (91 vs 74%; P < .05) with similar actionability (42 vs 34%; P = .15) when compared to the standard materials.

Conclusion: Patient-facing handouts can be augmented by ChatGPT with simple prompting to tailor information with improved readability. This study demonstrates the utility of LLMs to aid in rewriting patient handouts and may serve as a tool to help optimize education materials.

Level of evidence: Level VI.

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
提高健康素养:评估经 ChatGPT 大语言模型修订的患者手册的可读性。
目的:使用人工智能(AI)驱动的大型语言模型(LLM)提高患者手册的可读性:使用人工智能(AI)驱动的大型语言模型(LLM)提高患者手册的可读性:研究设计:审查经人工智能修改的在线资料:学术中心:使用经过验证的可读性指标对从美国鼻科学会(ARS)和美国面部整形外科学会网站上获取的五份讲义进行评估。根据提示将讲义输入到 OpenAI 的 ChatGPT-4 中:"以六年级的阅读水平重写以下内容"。使用患者教育材料评估工具(PEMAT)对原生版本和 LLM 修订版本的可理解性和可操作性进行了评估。结果采用 Wilcoxon 秩和检验进行比较:结果:标准版(ARS,美国面部整形与重建外科学会)材料的平均可读性评分为 "困难",阅读类别介于高中和大学年级之间。相反,LLM 修订版讲义的平均阅读水平为七年级水平。在几乎所有测试指标中,LLM 修订版讲义的可读性都更好:Flesch-Kincaid 阅读轻松度(70.8 vs 43.9;P 结论:LLM 修订版讲义的可读性更好:面向患者的讲义可以通过 ChatGPT 进行扩充,并通过简单的提示来定制信息,从而提高可读性。这项研究证明了 LLMs 在帮助改写患者手册方面的实用性,可作为帮助优化教育材料的工具:证据等级:VI 级。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Otolaryngology- Head and Neck Surgery
Otolaryngology- Head and Neck Surgery 医学-耳鼻喉科学
CiteScore
6.70
自引率
2.90%
发文量
250
审稿时长
2-4 weeks
期刊介绍: Otolaryngology–Head and Neck Surgery (OTO-HNS) is the official peer-reviewed publication of the American Academy of Otolaryngology–Head and Neck Surgery Foundation. The mission of Otolaryngology–Head and Neck Surgery is to publish contemporary, ethical, clinically relevant information in otolaryngology, head and neck surgery (ear, nose, throat, head, and neck disorders) that can be used by otolaryngologists, clinicians, scientists, and specialists to improve patient care and public health.
期刊最新文献
Selective Adipose Cryolysis for Reduction of Lingual Tissue in a Porcine Model. Tracheoesophageal Puncture Outcomes at a Safety Net Hospital. Author Reply to Letter by Kezirian Regarding Combination Tonsillectomy and Hypoglossal Nerve Stimulation for Sleep Apnea Patients With Oropharyngeal Lateral Wall Collapse. Comparative Analysis of Vestibular Dysfunction and Compensation in Ramsay-Hunt Syndrome and Vestibular Neuritis. Development and Validation of an Explainable Prediction Model for Postoperative Recurrence in Pediatric Chronic Rhinosinusitis.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1