Assessing the reliability and usefulness of ChatGPT responses on intermittent catheterization queries: A critical analysis

Abstract

The rapid development of technology has caused patients to start using artificial intelligence based applications more frequently to learn about their diseases. The aim of this study was to evaluate the reliability and usefulness of responses to questions related to intermittent catheterization using the large language model defined as ChatGPT. Ten questions were created based on the most searched keywords on Google Trends, the most searched questions on ChatGPT and the clinical experiences of the authors. The created questions were asked to ChatGPT and the answers were evaluated by five raters with seven-point Likert-type reliability and usefulness scales. The highest score in terms of reliability and usefulness was the response to “Is there a difference between the catheters used for intermittent catheterization in men and women?”(mean 6.4 ± 0.8; 6.0 ± 1.0). The lowest average reliability and usability score was for the response to “What is intermittent catheterization and how is it performed?”(mean 2.8 ± 0.8; 2.8 ± 0.8). ChatGPT is relatively very reliable and moderately useful in providing patients with information about intermittent catheterization, but it may give incorrect and misleading answers. It should be remembered that simple errors in intermittent catheterization can lead to major problems.