Chat GPT 4o vs residents: French language evaluation in ophthalmology

Abstract

Purpose

Chatbots capable of answering multiple-choice questions (MCQs) at a level comparable to residents could serve as affordable, 24/7 available educational tools with comprehensive explanations. Their non-judgmental nature could enable residents to freely ask questions without hesitation. Therefore, this study's aim is to evaluate ChatGPT 4o's accuracy to MCQs from the national ophthalmology residency examination in French language, compared to residents and other leading AI chatbots

Methods

A set of 600 questions from the national ophthalmology examination was translated into French and submitted to ChatGPT 4o, ChatGPT 4, and Gemini Advanced. The generated responses were compared to official correction grids to evaluate their accuracy. Additionally, variations over time, specialties, and accuracy with both text-based and image-based questions were analysed and compared to residents’ results.

Results

ChatGPT 4o achieved an accuracy rate of 67.5 %, outperforming the accuracy of ChatGPT 4 and Gemini Advanced. However, Gemini Advanced exhibited greater sensitivity to the ethical considerations involved in medical advice generation. ChatGPT 4o demonstrated consistent accuracy over time, with particular strength in the fundamentals of ophthalmology, ocular pathologies, and refractive surgery. Its performance in image processing was significantly improved compared to other chatbots, though still inferior to text-based processing.

Conclusion

ChatGPT 4o demonstrates sufficient accuracy to pass the ophthalmology national examination, though its performance falls short compared to that of residents. These findings suggest that the use of ChatGPT 4o as an educational tool in ophthalmology residency is promising, even in a non-English language. However, further improvements are needed to enhance its performances.