Are AI chatbots concordant with evidence-based cancer screening recommendations?

IF 2.9 2区医学 Q2 PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH Patient Education and Counseling Pub Date : 2025-01-21 DOI:10.1016/j.pec.2025.108677

Brooke Nickel , Julie Ayre , M Luke Marinovich , David P. Smith , Karen Chiam , Christoph I. Lee , Timothy J. Wilt , Melody Taba , Kirsten McCaffery , Nehmat Houssami

{"title":"Are AI chatbots concordant with evidence-based cancer screening recommendations?","authors":"Brooke Nickel , Julie Ayre , M Luke Marinovich , David P. Smith , Karen Chiam , Christoph I. Lee , Timothy J. Wilt , Melody Taba , Kirsten McCaffery , Nehmat Houssami","doi":"10.1016/j.pec.2025.108677","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>This study aimed to assess whether information from AI chatbots on benefits and harms of breast and prostate cancer screening were concordant with evidence-based cancer screening recommendations.</div></div><div><h3>Methods</h3><div>Seven unique prompts (four breast cancer; three prostate cancer) were presented to ChatGPT in March 2024. A total of 60 criteria (30 breast; 30 prostate) were used to assess the concordance of information. Concordance was scored between 0 and 2 against the United States Preventive Services Task Force (USPSTF) breast and prostate cancer screening recommendations independently by international cancer screening experts.</div></div><div><h3>Results</h3><div>43 of 60 (71.7 %) criteria were completely concordant, 3 (5 %) were moderately concordant and 14 (23.3 %) were not concordant or not present, with most of the non-concordant criteria (9 of 14, 64.3 %) being from prompts for the oldest age groups. ChatGPT hallucinations (i.e., completely made up, non-sensical or irrelevant information) were found in 9 of 60 criteria (15 %).</div></div><div><h3>Conclusions</h3><div>ChatGPT provided information mostly concordant with USPSTF breast and prostate cancer screening recommendations, however, important gaps exist. These findings provide insights into the role of AI to communicate cancer screening benefits and harms and hold increased relevance for periods of guideline change.</div></div><div><h3>Practice implications</h3><div>AI generated information on cancer screening should be taken in conjunction with official screening recommendations and/or information from clinicians.</div></div>","PeriodicalId":49714,"journal":{"name":"Patient Education and Counseling","volume":"134 ","pages":"Article 108677"},"PeriodicalIF":2.9000,"publicationDate":"2025-01-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Patient Education and Counseling","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0738399125000448","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

This study aimed to assess whether information from AI chatbots on benefits and harms of breast and prostate cancer screening were concordant with evidence-based cancer screening recommendations.

Methods

Seven unique prompts (four breast cancer; three prostate cancer) were presented to ChatGPT in March 2024. A total of 60 criteria (30 breast; 30 prostate) were used to assess the concordance of information. Concordance was scored between 0 and 2 against the United States Preventive Services Task Force (USPSTF) breast and prostate cancer screening recommendations independently by international cancer screening experts.

Results

43 of 60 (71.7 %) criteria were completely concordant, 3 (5 %) were moderately concordant and 14 (23.3 %) were not concordant or not present, with most of the non-concordant criteria (9 of 14, 64.3 %) being from prompts for the oldest age groups. ChatGPT hallucinations (i.e., completely made up, non-sensical or irrelevant information) were found in 9 of 60 criteria (15 %).

Conclusions

ChatGPT provided information mostly concordant with USPSTF breast and prostate cancer screening recommendations, however, important gaps exist. These findings provide insights into the role of AI to communicate cancer screening benefits and harms and hold increased relevance for periods of guideline change.

Practice implications

AI generated information on cancer screening should be taken in conjunction with official screening recommendations and/or information from clinicians.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

求助全文

约1分钟内获得全文去求助

来源期刊

Patient Education and Counseling 医学-公共卫生、环境卫生与职业卫生

CiteScore

5.60

自引率

11.40%

发文量

384

审稿时长

46 days

期刊介绍： Patient Education and Counseling is an interdisciplinary, international journal for patient education and health promotion researchers, managers and clinicians. The journal seeks to explore and elucidate the educational, counseling and communication models in health care. Its aim is to provide a forum for fundamental as well as applied research, and to promote the study of organizational issues involved with the delivery of patient education, counseling, health promotion services and training models in improving communication between providers and patients.