Assessing the performance of Microsoft Copilot, GPT-4 and Google Gemini in ophthalmology.

Meziane Silhadi, Wissam B Nassrallah, David Mikhail, Daniel Milad, Mona Harissi-Dagher
{"title":"Assessing the performance of Microsoft Copilot, GPT-4 and Google Gemini in ophthalmology.","authors":"Meziane Silhadi, Wissam B Nassrallah, David Mikhail, Daniel Milad, Mona Harissi-Dagher","doi":"10.1016/j.jcjo.2025.01.001","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>To evaluate the performance of large language models (LLMs), specifically Microsoft Copilot, GPT-4 (GPT-4o and GPT-4o mini), and Google Gemini (Gemini and Gemini Advanced), in answering ophthalmological questions and assessing the impact of prompting techniques on their accuracy.</p><p><strong>Design: </strong>Prospective qualitative study.</p><p><strong>Participants: </strong>Microsoft Copilot, GPT-4 (GPT-4o and GPT-4o mini), and Google Gemini (Gemini and Gemini Advanced).</p><p><strong>Methods: </strong>A total of 300 ophthalmological questions from StatPearls were tested, covering a range of subspecialties and image-based tasks. Each question was evaluated using 2 prompting techniques: zero-shot forced prompting (prompt 1) and combined role-based and zero-shot plan-and-solve+ prompting (prompt 2).</p><p><strong>Results: </strong>With zero-shot forced prompting, GPT-4o demonstrated significantly superior overall performance, correctly answering 72.3% of questions and outperforming all other models, including Copilot (53.7%), GPT-4o mini (62.0%), Gemini (54.3%), and Gemini Advanced (62.0%) (p < 0.0001). Both Copilot and GPT-4o showed notable improvements with Prompt 2 over Prompt 1, elevating Copilot's accuracy from the lowest (53.7%) to the second highest (72.3%) among the evaluated LLMs.</p><p><strong>Conclusions: </strong>While newer iterations of LLMs, such as GPT-4o and Gemini Advanced, outperformed their less advanced counterparts (GPT-4o mini and Gemini), this study emphasizes the need for caution in clinical applications of these models. The choice of prompting techniques significantly influences performance, highlighting the necessity for further research to refine LLMs capabilities, particularly in visual data interpretation, to ensure their safe integration into medical practice.</p>","PeriodicalId":9606,"journal":{"name":"Canadian journal of ophthalmology. Journal canadien d'ophtalmologie","volume":" ","pages":""},"PeriodicalIF":3.3000,"publicationDate":"2025-02-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Canadian journal of ophthalmology. Journal canadien d'ophtalmologie","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.jcjo.2025.01.001","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"OPHTHALMOLOGY","Score":null,"Total":0}
引用次数: 0

Abstract

Objective: To evaluate the performance of large language models (LLMs), specifically Microsoft Copilot, GPT-4 (GPT-4o and GPT-4o mini), and Google Gemini (Gemini and Gemini Advanced), in answering ophthalmological questions and assessing the impact of prompting techniques on their accuracy.

Design: Prospective qualitative study.

Participants: Microsoft Copilot, GPT-4 (GPT-4o and GPT-4o mini), and Google Gemini (Gemini and Gemini Advanced).

Methods: A total of 300 ophthalmological questions from StatPearls were tested, covering a range of subspecialties and image-based tasks. Each question was evaluated using 2 prompting techniques: zero-shot forced prompting (prompt 1) and combined role-based and zero-shot plan-and-solve+ prompting (prompt 2).

Results: With zero-shot forced prompting, GPT-4o demonstrated significantly superior overall performance, correctly answering 72.3% of questions and outperforming all other models, including Copilot (53.7%), GPT-4o mini (62.0%), Gemini (54.3%), and Gemini Advanced (62.0%) (p < 0.0001). Both Copilot and GPT-4o showed notable improvements with Prompt 2 over Prompt 1, elevating Copilot's accuracy from the lowest (53.7%) to the second highest (72.3%) among the evaluated LLMs.

Conclusions: While newer iterations of LLMs, such as GPT-4o and Gemini Advanced, outperformed their less advanced counterparts (GPT-4o mini and Gemini), this study emphasizes the need for caution in clinical applications of these models. The choice of prompting techniques significantly influences performance, highlighting the necessity for further research to refine LLMs capabilities, particularly in visual data interpretation, to ensure their safe integration into medical practice.

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
评估 Microsoft Copilot、GPT-4 和 Google Gemini 在眼科方面的性能。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
CiteScore
3.20
自引率
4.80%
发文量
223
审稿时长
38 days
期刊介绍: Official journal of the Canadian Ophthalmological Society. The Canadian Journal of Ophthalmology (CJO) is the official journal of the Canadian Ophthalmological Society and is committed to timely publication of original, peer-reviewed ophthalmology and vision science articles.
期刊最新文献
Accuracy of keratoconus-specific formulae compared to standard formulae for intraocular lens power calculation in patients with keratoconus. Optic nerve sheath fenestration for young patients with papilledema secondary to cerebral sinovenous thrombosis: a case series. Effect of implementation of an electronic consult referral platform (eConsult) to triage retina referrals in Manitoba. Visual outcomes and safety profile of retropupillary iris-claw Artisan intraocular lens implants. Sensitivity of ophthalmologists, residents, and optometrists in identifying peripheral retinal tears on ultra-widefield imaging.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1