Utility of Chatbot Literature Search in Radiation Oncology.

Abstract

Artificial intelligence and natural language processing tools have shown promise in oncology by assisting with medical literature retrieval and providing patient support. The potential for these technologies to generate inaccurate yet seemingly correct information poses significant challenges. This study evaluates the effectiveness, benefits, and limitations of ChatGPT for clinical use in conducting literature reviews of radiation oncology treatments. This cross-sectional study used ChatGPT version 3.5 to generate literature searches on radiotherapy options for seven tumor sites, with prompts issued five times per site to generate up to 50 publications per tumor type. The publications were verified using the Scopus database and categorized as correct, irrelevant, or non-existent. Statistical analysis with one-way ANOVA compared the impact factors and citation counts across different tumor sites. Among the 350 publications generated, there were 44 correct, 298 non-existent, and 8 irrelevant papers. The average publication year of all generated papers was 2011, compared to 2009 for the correct papers. The average impact factor of all generated papers was 38.8, compared to 113.8 for the correct papers. There were significant differences in the publication year, impact factor, and citation counts between tumor sites for both correct and non-existent papers. Our study highlights both the potential utility and significant limitations of using AI, specifically ChatGPT 3.5, in radiation oncology literature reviews. The findings emphasize the need for verification of AI outputs, development of standardized quality assurance protocols, and continued research into AI biases to ensure reliable integration into clinical practice.