Evaluation of Responses to Questions About Keratoconus Using ChatGPT-4.0, Google Gemini and Microsoft Copilot: A Comparative Study of Large Language Models on Keratoconus.

Abstract

Objectives: Large language models (LLMs) are increasingly being used today and are becoming increasingly important for providing accurate clinical information to patients and physicians. This study aimed to evaluate the effectiveness of generative pre-trained transforme-4.0 (ChatGPT-4.0), Google Gemini, and Microsoft Copilot LLMs in responding to patient questions regarding keratoconus.

Methods: The LLMs' responses to the 25 most common questions about keratoconus asked by real-life patients were blindly rated by two ophthalmologists using a 5-point Likert scale. In addition, the DISCERN scale was used to evaluate the responses of the language models in terms of reliability, and the Flesch reading ease and Flesch-Kincaid grade level indices were used to determine readability.

Results: ChatGPT-4.0 provided more detailed and accurate answers to patients' questions about keratoconus than Google Gemini and Microsoft Copilot, with 92% of the answers belonging to the "agree" or "strongly agree" categories. Significant differences were observed between all three LLMs on the Likert scale (P<0.001).

Conclusions: Although the answers of ChatGPT-4.0 to questions about keratoconus were more complex for patients than those of other language programs, the information provided was reliable and accurate.