Are AI chatbots concordant with evidence-based cancer screening recommendations?

IF 3.1 2区医学 Q2 PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH Patient Education and Counseling Pub Date : 2025-05-01 Epub Date: 2025-01-21 DOI:10.1016/j.pec.2025.108677

Brooke Nickel , Julie Ayre , M Luke Marinovich , David P. Smith , Karen Chiam , Christoph I. Lee , Timothy J. Wilt , Melody Taba , Kirsten McCaffery , Nehmat Houssami

{"title":"Are AI chatbots concordant with evidence-based cancer screening recommendations?","authors":"Brooke Nickel , Julie Ayre , M Luke Marinovich , David P. Smith , Karen Chiam , Christoph I. Lee , Timothy J. Wilt , Melody Taba , Kirsten McCaffery , Nehmat Houssami","doi":"10.1016/j.pec.2025.108677","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>This study aimed to assess whether information from AI chatbots on benefits and harms of breast and prostate cancer screening were concordant with evidence-based cancer screening recommendations.</div></div><div><h3>Methods</h3><div>Seven unique prompts (four breast cancer; three prostate cancer) were presented to ChatGPT in March 2024. A total of 60 criteria (30 breast; 30 prostate) were used to assess the concordance of information. Concordance was scored between 0 and 2 against the United States Preventive Services Task Force (USPSTF) breast and prostate cancer screening recommendations independently by international cancer screening experts.</div></div><div><h3>Results</h3><div>43 of 60 (71.7 %) criteria were completely concordant, 3 (5 %) were moderately concordant and 14 (23.3 %) were not concordant or not present, with most of the non-concordant criteria (9 of 14, 64.3 %) being from prompts for the oldest age groups. ChatGPT hallucinations (i.e., completely made up, non-sensical or irrelevant information) were found in 9 of 60 criteria (15 %).</div></div><div><h3>Conclusions</h3><div>ChatGPT provided information mostly concordant with USPSTF breast and prostate cancer screening recommendations, however, important gaps exist. These findings provide insights into the role of AI to communicate cancer screening benefits and harms and hold increased relevance for periods of guideline change.</div></div><div><h3>Practice implications</h3><div>AI generated information on cancer screening should be taken in conjunction with official screening recommendations and/or information from clinicians.</div></div>","PeriodicalId":49714,"journal":{"name":"Patient Education and Counseling","volume":"134 ","pages":"Article 108677"},"PeriodicalIF":3.1000,"publicationDate":"2025-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Patient Education and Counseling","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0738399125000448","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/1/21 0:00:00","PubModel":"Epub","JCR":"Q2","JCRName":"PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

This study aimed to assess whether information from AI chatbots on benefits and harms of breast and prostate cancer screening were concordant with evidence-based cancer screening recommendations.

Methods

Seven unique prompts (four breast cancer; three prostate cancer) were presented to ChatGPT in March 2024. A total of 60 criteria (30 breast; 30 prostate) were used to assess the concordance of information. Concordance was scored between 0 and 2 against the United States Preventive Services Task Force (USPSTF) breast and prostate cancer screening recommendations independently by international cancer screening experts.

Results

43 of 60 (71.7 %) criteria were completely concordant, 3 (5 %) were moderately concordant and 14 (23.3 %) were not concordant or not present, with most of the non-concordant criteria (9 of 14, 64.3 %) being from prompts for the oldest age groups. ChatGPT hallucinations (i.e., completely made up, non-sensical or irrelevant information) were found in 9 of 60 criteria (15 %).

Conclusions

ChatGPT provided information mostly concordant with USPSTF breast and prostate cancer screening recommendations, however, important gaps exist. These findings provide insights into the role of AI to communicate cancer screening benefits and harms and hold increased relevance for periods of guideline change.

Practice implications

AI generated information on cancer screening should be taken in conjunction with official screening recommendations and/or information from clinicians.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

人工智能聊天机器人是否符合循证癌症筛查建议？

目的：本研究旨在评估人工智能聊天机器人关于乳腺癌和前列腺癌筛查的利弊信息是否与循证癌症筛查建议一致。方法：7种独特提示(4种乳腺癌；3例前列腺癌)于2024年3月提交给ChatGPT。共60项标准(乳腺30项；30例前列腺癌患者)评估信息的一致性。根据国际癌症筛查专家独立提出的美国预防服务工作组（USPSTF）乳腺癌和前列腺癌筛查建议，一致性评分在0到2之间。结果：60条标准中43条（71.7 %）完全符合，3条（5 %）中度符合，14条（23.3 %）不符合或不存在，其中大部分不符合标准（14条中9条，64.3%）来自年龄较大年龄组的提示。ChatGPT幻觉（即，完全编造，无意义或不相关的信息）在60个标准中的9个中被发现（15% %）。结论：ChatGPT提供的信息与USPSTF乳腺癌和前列腺癌筛查建议基本一致，但存在重要差距。这些发现为人工智能在传达癌症筛查益处和危害方面的作用提供了见解，并与指南变化时期具有更大的相关性。实践意义：人工智能产生的癌症筛查信息应与官方筛查建议和/或临床医生提供的信息结合使用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Patient Education and Counseling 医学-公共卫生、环境卫生与职业卫生

CiteScore

5.60

自引率

11.40%

发文量

384

审稿时长

46 days

期刊介绍： Patient Education and Counseling is an interdisciplinary, international journal for patient education and health promotion researchers, managers and clinicians. The journal seeks to explore and elucidate the educational, counseling and communication models in health care. Its aim is to provide a forum for fundamental as well as applied research, and to promote the study of organizational issues involved with the delivery of patient education, counseling, health promotion services and training models in improving communication between providers and patients.