Evaluation of ChatGPT-4 Performance in Answering Patients' Questions About the Management of Type 2 Diabetes.

Abstract

Objectives: Type 2 diabetes mellitus is a disease with a rising prevalence worldwide. Person-centered treatment factors, including comorbidities and treatment goals, should be considered in determining the pharmacological treatment of type 2 diabetes. ChatGPT-4 (Generative Pre-trained Transformer), a large language model, holds the potential performance in various fields, including medicine. We aimed to examine the reliability, quality, reproducibility, and readability of ChatGPT-4's responses to clinical scenarios about the medical treatment approach and management of type 2 diabetes patients.

Methods: ChatGPT-4's responses to 24 questions were independently graded by two endocrinologists with clinical experience in endocrinology and resolved by a third reviewer based on the ADA(American Diabetes Association) 2023 guidelines. DISCERN (Quality Criteria for Consumer Health Information) Measurement Tool was used to evaluate the reliability and quality of information.

Results: Responses to questions by ChatGPT-4 were fairly consistent in both sessions. No false or misleading information was found in any ChatGPT-4 responses. In terms of reliability, most of the answers showed good (87.5%), followed by excellent (12.5%) reliability. Reading Level was classified as fairly difficult to read (8.3%), difficult to read (50%), and very difficult to read (41.7%).

Conclusion: ChatGPT-4 may have a role as an additional informative tool for type 2 diabetes patients for medical treatment approaches.