Performance of GPT-4V(ision) in Ophthalmology: Use of Images in Clinical Questions

Background/aims: To compare the diagnostic accuracy of Generative Pre-trained Transformer with Vision (GPT)-4 and GPT-4 with Vision (GPT-4V) for clinical questions in ophthalmology. Methods: The questions were collected from the "Diagnosis This" section on the American Academy of Ophthalmology website. We tested 580 questions and presented GPT-4V with the same questions under two conditions: 1) multimodal model, incorporating both the question text and associated images, and 2) text-only model. We then compared the difference in accuracy between the two conditions using the chi-square test. The percentage of general correct answers was also collected from the website. Results: The GPT-4V model demonstrated higher accuracy with images (71.7%) than without images (66.7%, p<0.001). Both GPT-4 models showed higher accuracy than the general correct answers on the website [64.6 (95%CI, 62.9 to 66.3)]. Conclusions: The addition of information from images enhances the performance of GPT-4V in diagnosing clinical questions in ophthalmology. This suggests that integrating multimodal data could be crucial in developing more effective and reliable diagnostic tools in medical fields.