评估 ChatGPT 在常见患者关于 HPV+ 口咽癌问题中的准确性。

IF 1.3 4区 医学 Q3 OTORHINOLARYNGOLOGY Annals of Otology Rhinology and Laryngology Pub Date : 2024-09-01 Epub Date: 2024-07-29 DOI:10.1177/00034894241259137
Nikhil Bellamkonda, Janice L Farlow, Catherine T Haring, Michael W Sim, Nolan B Seim, Richard B Cannon, Marcus M Monroe, Amit Agrawal, James W Rocco, Hilary C McCrary
{"title":"评估 ChatGPT 在常见患者关于 HPV+ 口咽癌问题中的准确性。","authors":"Nikhil Bellamkonda, Janice L Farlow, Catherine T Haring, Michael W Sim, Nolan B Seim, Richard B Cannon, Marcus M Monroe, Amit Agrawal, James W Rocco, Hilary C McCrary","doi":"10.1177/00034894241259137","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Large language model (LLM)-based chatbots such as ChatGPT have been publicly available and increasingly utilized by the general public since late 2022. This study sought to investigate ChatGPT responses to common patient questions regarding Human Papilloma Virus (HPV) positive oropharyngeal cancer (OPC).</p><p><strong>Methods: </strong>This was a prospective, multi-institutional study, with data collected from high volume institutions that perform >50 transoral robotic surgery cases per year. The 100 most recent discussion threads including the term \"HPV\" on the American Cancer Society's Cancer Survivors Network's Head and Neck Cancer public discussion board were reviewed. The 11 most common questions were serially queried to ChatGPT 3.5; answers were recorded. A survey was distributed to fellowship trained head and neck oncologic surgeons at 3 institutions to evaluate the responses.</p><p><strong>Results: </strong>A total of 8 surgeons participated in the study. For questions regarding HPV contraction and transmission, ChatGPT answers were scored as clinically accurate and aligned with consensus in the head and neck surgical oncology community 84.4% and 90.6% of the time, respectively. For questions involving treatment of HPV+ OPC, ChatGPT was clinically accurate and aligned with consensus 87.5% and 91.7% of the time, respectively. For questions regarding the HPV vaccine, ChatGPT was clinically accurate and aligned with consensus 62.5% and 75% of the time, respectively. When asked about circulating tumor DNA testing, only 12.5% of surgeons thought responses were accurate or consistent with consensus.</p><p><strong>Conclusion: </strong>ChatGPT 3.5 performed poorly with questions involving evolving therapies and diagnostics-thus, caution should be used when using a platform like ChatGPT 3.5 to assess use of advanced technology. Patients should be counseled on the importance of consulting their surgeons to receive accurate and up to date recommendations, and use LLM's to augment their understanding of these important health-related topics.</p>","PeriodicalId":50975,"journal":{"name":"Annals of Otology Rhinology and Laryngology","volume":null,"pages":null},"PeriodicalIF":1.3000,"publicationDate":"2024-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Evaluating the Accuracy of ChatGPT in Common Patient Questions Regarding HPV+ Oropharyngeal Carcinoma.\",\"authors\":\"Nikhil Bellamkonda, Janice L Farlow, Catherine T Haring, Michael W Sim, Nolan B Seim, Richard B Cannon, Marcus M Monroe, Amit Agrawal, James W Rocco, Hilary C McCrary\",\"doi\":\"10.1177/00034894241259137\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Objectives: </strong>Large language model (LLM)-based chatbots such as ChatGPT have been publicly available and increasingly utilized by the general public since late 2022. This study sought to investigate ChatGPT responses to common patient questions regarding Human Papilloma Virus (HPV) positive oropharyngeal cancer (OPC).</p><p><strong>Methods: </strong>This was a prospective, multi-institutional study, with data collected from high volume institutions that perform >50 transoral robotic surgery cases per year. The 100 most recent discussion threads including the term \\\"HPV\\\" on the American Cancer Society's Cancer Survivors Network's Head and Neck Cancer public discussion board were reviewed. The 11 most common questions were serially queried to ChatGPT 3.5; answers were recorded. A survey was distributed to fellowship trained head and neck oncologic surgeons at 3 institutions to evaluate the responses.</p><p><strong>Results: </strong>A total of 8 surgeons participated in the study. For questions regarding HPV contraction and transmission, ChatGPT answers were scored as clinically accurate and aligned with consensus in the head and neck surgical oncology community 84.4% and 90.6% of the time, respectively. For questions involving treatment of HPV+ OPC, ChatGPT was clinically accurate and aligned with consensus 87.5% and 91.7% of the time, respectively. For questions regarding the HPV vaccine, ChatGPT was clinically accurate and aligned with consensus 62.5% and 75% of the time, respectively. When asked about circulating tumor DNA testing, only 12.5% of surgeons thought responses were accurate or consistent with consensus.</p><p><strong>Conclusion: </strong>ChatGPT 3.5 performed poorly with questions involving evolving therapies and diagnostics-thus, caution should be used when using a platform like ChatGPT 3.5 to assess use of advanced technology. Patients should be counseled on the importance of consulting their surgeons to receive accurate and up to date recommendations, and use LLM's to augment their understanding of these important health-related topics.</p>\",\"PeriodicalId\":50975,\"journal\":{\"name\":\"Annals of Otology Rhinology and Laryngology\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":1.3000,\"publicationDate\":\"2024-09-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Annals of Otology Rhinology and Laryngology\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://doi.org/10.1177/00034894241259137\",\"RegionNum\":4,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2024/7/29 0:00:00\",\"PubModel\":\"Epub\",\"JCR\":\"Q3\",\"JCRName\":\"OTORHINOLARYNGOLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Annals of Otology Rhinology and Laryngology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1177/00034894241259137","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/7/29 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"OTORHINOLARYNGOLOGY","Score":null,"Total":0}
引用次数: 0

摘要

目的:基于大语言模型(LLM)的聊天机器人,如 ChatGPT,自 2022 年末以来已经公开可用,并越来越多地被公众使用。本研究旨在调查 ChatGPT 对患者关于人乳头状瘤病毒(HPV)阳性口咽癌(OPC)常见问题的回答:这是一项前瞻性、多机构研究,数据收集自每年经口机器人手术例数大于 50 例的大医院。研究人员查阅了美国癌症协会癌症幸存者网络头颈癌公共讨论板上包含 "HPV "一词的100条最新讨论主题。在 ChatGPT 3.5 中连续查询了 11 个最常见的问题,并记录了答案。向 3 家机构受过研究培训的头颈部肿瘤外科医生发放了调查问卷,以评估回复情况:共有 8 名外科医生参与了这项研究。对于有关 HPV 感染和传播的问题,ChatGPT 的答案被评为临床准确,并分别有 84.4% 和 90.6% 的时间与头颈部肿瘤外科界的共识一致。对于涉及 HPV+ OPC 治疗的问题,ChatGPT 的临床准确性和符合共识的比例分别为 87.5% 和 91.7%。对于有关 HPV 疫苗的问题,ChatGPT 的临床准确性和与共识一致的比例分别为 62.5% 和 75%。当被问及循环肿瘤 DNA 检测时,只有 12.5% 的外科医生认为回答准确或符合共识:ChatGPT 3.5 在涉及不断发展的疗法和诊断的问题上表现不佳,因此在使用 ChatGPT 3.5 这样的平台评估先进技术的使用情况时应谨慎。应建议患者咨询他们的外科医生,以获得准确和最新的建议,并使用 LLM 增强他们对这些重要健康相关主题的了解。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
Evaluating the Accuracy of ChatGPT in Common Patient Questions Regarding HPV+ Oropharyngeal Carcinoma.

Objectives: Large language model (LLM)-based chatbots such as ChatGPT have been publicly available and increasingly utilized by the general public since late 2022. This study sought to investigate ChatGPT responses to common patient questions regarding Human Papilloma Virus (HPV) positive oropharyngeal cancer (OPC).

Methods: This was a prospective, multi-institutional study, with data collected from high volume institutions that perform >50 transoral robotic surgery cases per year. The 100 most recent discussion threads including the term "HPV" on the American Cancer Society's Cancer Survivors Network's Head and Neck Cancer public discussion board were reviewed. The 11 most common questions were serially queried to ChatGPT 3.5; answers were recorded. A survey was distributed to fellowship trained head and neck oncologic surgeons at 3 institutions to evaluate the responses.

Results: A total of 8 surgeons participated in the study. For questions regarding HPV contraction and transmission, ChatGPT answers were scored as clinically accurate and aligned with consensus in the head and neck surgical oncology community 84.4% and 90.6% of the time, respectively. For questions involving treatment of HPV+ OPC, ChatGPT was clinically accurate and aligned with consensus 87.5% and 91.7% of the time, respectively. For questions regarding the HPV vaccine, ChatGPT was clinically accurate and aligned with consensus 62.5% and 75% of the time, respectively. When asked about circulating tumor DNA testing, only 12.5% of surgeons thought responses were accurate or consistent with consensus.

Conclusion: ChatGPT 3.5 performed poorly with questions involving evolving therapies and diagnostics-thus, caution should be used when using a platform like ChatGPT 3.5 to assess use of advanced technology. Patients should be counseled on the importance of consulting their surgeons to receive accurate and up to date recommendations, and use LLM's to augment their understanding of these important health-related topics.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
CiteScore
3.10
自引率
7.10%
发文量
171
审稿时长
4-8 weeks
期刊介绍: The Annals of Otology, Rhinology & Laryngology publishes original manuscripts of clinical and research importance in otolaryngology–head and neck medicine and surgery, otology, neurotology, bronchoesophagology, laryngology, rhinology, head and neck oncology and surgery, plastic and reconstructive surgery, pediatric otolaryngology, audiology, and speech pathology. In-depth studies (supplements), papers of historical interest, and reviews of computer software and applications in otolaryngology are also published, as well as imaging, pathology, and clinicopathology studies, book reviews, and letters to the editor. AOR is the official journal of the American Broncho-Esophagological Association.
期刊最新文献
Smartphone-Based Cognitive Behavioral Therapy and Customized Sound Therapy for Tinnitus: A Randomized Controlled Trial. The Effect of Rosuvastatin on Facial Nerve Regeneration After Facial Nerve Injury: An Experimental Animal Study. Assessing the Efficacy of Acupuncture in the Treatment of Olfactory Dysfunction: A Systematic Review. Letter to the Editor: "Comparative Analysis of Nebulized Versus Intravenous Fentanyl for Pain Control After Tonsillectomy: A Double-Blind, Randomized, Controlled Trial". The Impact of COVID-19 and Socioeconomic Determinants on Appointment Non-Attendance in an Urban Otolaryngology Clinic: A Retrospective Analysis From a Safety Net Hospital.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1