Assessing ChatGPT's Capability in Addressing Thyroid Cancer Patient Queries: A Comprehensive Mixed-Methods Evaluation.

IF 3 Q2 ENDOCRINOLOGY & METABOLISM Journal of the Endocrine Society Pub Date : 2025-01-13 eCollection Date: 2025-01-06 DOI:10.1210/jendso/bvaf003

Matthew A Gorris, Reese W Randle, Corey S Obermiller, Johnson Thomas, David Toro-Tobon, Sophie Y Dream, Oliver J Fackelmayer, T K Pandian, Sarah E Mayson

{"title":"Assessing ChatGPT's Capability in Addressing Thyroid Cancer Patient Queries: A Comprehensive Mixed-Methods Evaluation.","authors":"Matthew A Gorris, Reese W Randle, Corey S Obermiller, Johnson Thomas, David Toro-Tobon, Sophie Y Dream, Oliver J Fackelmayer, T K Pandian, Sarah E Mayson","doi":"10.1210/jendso/bvaf003","DOIUrl":null,"url":null,"abstract":"Context: Literature suggests patients with thyroid cancer have unmet informational needs in many aspects of care. Patients often turn to online resources for their health-related information, and generative artificial intelligence programs such as ChatGPT are an emerging and attractive resource for patients.Objective: To assess the quality of ChatGPT's responses to thyroid cancer-related questions.Methods: Four endocrinologists and 4 endocrine surgeons, all with expertise in thyroid cancer, evaluated the responses to 20 thyroid cancer-related questions. Responses were scored on a 7-point Likert scale in areas of accuracy, completeness, and overall satisfaction. Comments from the evaluators were aggregated and a qualitative analysis was performed.Results: Overall, only 57%, 56%, and 52% of the responses \"agreed\" or \"strongly agreed\" that ChatGPT's answers were accurate, complete, and satisfactory, respectively. One hundred ninety-eight free-text comments were included in the qualitative analysis. The majority of comments were critical in nature. Several themes emerged, which included overemphasis of diet and iodine intake and its role in thyroid cancer, and incomplete or inaccurate information on risks of both thyroid surgery and radioactive iodine therapy.Conclusion: Our study suggests that ChatGPT is not accurate or reliable enough at this time for unsupervised use as a patient information tool for thyroid cancer.","PeriodicalId":17334,"journal":{"name":"Journal of the Endocrine Society","volume":"9 2","pages":"bvaf003"},"PeriodicalIF":3.0000,"publicationDate":"2025-01-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11775116/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the Endocrine Society","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1210/jendso/bvaf003","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/1/6 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"ENDOCRINOLOGY & METABOLISM","Score":null,"Total":0}

引用次数: 0

Abstract

Context: Literature suggests patients with thyroid cancer have unmet informational needs in many aspects of care. Patients often turn to online resources for their health-related information, and generative artificial intelligence programs such as ChatGPT are an emerging and attractive resource for patients.

Objective: To assess the quality of ChatGPT's responses to thyroid cancer-related questions.

Methods: Four endocrinologists and 4 endocrine surgeons, all with expertise in thyroid cancer, evaluated the responses to 20 thyroid cancer-related questions. Responses were scored on a 7-point Likert scale in areas of accuracy, completeness, and overall satisfaction. Comments from the evaluators were aggregated and a qualitative analysis was performed.

Results: Overall, only 57%, 56%, and 52% of the responses "agreed" or "strongly agreed" that ChatGPT's answers were accurate, complete, and satisfactory, respectively. One hundred ninety-eight free-text comments were included in the qualitative analysis. The majority of comments were critical in nature. Several themes emerged, which included overemphasis of diet and iodine intake and its role in thyroid cancer, and incomplete or inaccurate information on risks of both thyroid surgery and radioactive iodine therapy.

Conclusion: Our study suggests that ChatGPT is not accurate or reliable enough at this time for unsupervised use as a patient information tool for thyroid cancer.

查看原文