The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination.

IF 1.1 4区 医学 Q3 SURGERY Annals of the Royal College of Surgeons of England Pub Date : 2024-11-01 Epub Date: 2024-03-06 DOI:10.1308/rcsann.2024.0023
J Chan, T Dong, G D Angelini
{"title":"The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination.","authors":"J Chan, T Dong, G D Angelini","doi":"10.1308/rcsann.2024.0023","DOIUrl":null,"url":null,"abstract":"<p><strong>Introduction: </strong>Large language models (LLM), such as Chat Generative Pre-trained Transformer (ChatGPT) and Bard utilise deep learning algorithms that have been trained on a massive data set of text and code to generate human-like responses. Several studies have demonstrated satisfactory performance on postgraduate examinations, including the United States Medical Licensing Examination. We aimed to evaluate artificial intelligence performance in Part A of the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination.</p><p><strong>Methods: </strong>The MRCS mock examination from Pastest, a commonly used question bank for examinees, was used to assess the performance of three LLMs: GPT-3.5, GPT 4.0 and Bard. Three hundred mock questions were input into the three LLMs, and the responses provided by the LLMs were recorded and analysed. The pass mark was set at 70%.</p><p><strong>Results: </strong>The overall accuracies for GPT-3.5, GPT 4.0 and Bard were 67.33%, 71.67% and 65.67%, respectively (<i>p</i> = 0.27). The performances of GPT-3.5, GPT 4.0 and Bard in Applied Basic Sciences were 68.89%, 72.78% and 63.33% (<i>p</i> = 0.15), respectively. Furthermore, the three LLMs obtained correct answers in 65.00%, 70.00% and 69.17% of the Principles of Surgery in General questions (<i>p</i> = 0.67). There were no differences in performance in the overall and subcategories among the three LLMs.</p><p><strong>Conclusions: </strong>Our findings demonstrated satisfactory performance for all three LLMs in the MRCS Part A examination, with GPT 4.0 the only LLM that achieved the pass mark set.</p>","PeriodicalId":8088,"journal":{"name":"Annals of the Royal College of Surgeons of England","volume":null,"pages":null},"PeriodicalIF":1.1000,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11528401/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Annals of the Royal College of Surgeons of England","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1308/rcsann.2024.0023","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/3/6 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"SURGERY","Score":null,"Total":0}
引用次数: 0

Abstract

Introduction: Large language models (LLM), such as Chat Generative Pre-trained Transformer (ChatGPT) and Bard utilise deep learning algorithms that have been trained on a massive data set of text and code to generate human-like responses. Several studies have demonstrated satisfactory performance on postgraduate examinations, including the United States Medical Licensing Examination. We aimed to evaluate artificial intelligence performance in Part A of the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination.

Methods: The MRCS mock examination from Pastest, a commonly used question bank for examinees, was used to assess the performance of three LLMs: GPT-3.5, GPT 4.0 and Bard. Three hundred mock questions were input into the three LLMs, and the responses provided by the LLMs were recorded and analysed. The pass mark was set at 70%.

Results: The overall accuracies for GPT-3.5, GPT 4.0 and Bard were 67.33%, 71.67% and 65.67%, respectively (p = 0.27). The performances of GPT-3.5, GPT 4.0 and Bard in Applied Basic Sciences were 68.89%, 72.78% and 63.33% (p = 0.15), respectively. Furthermore, the three LLMs obtained correct answers in 65.00%, 70.00% and 69.17% of the Principles of Surgery in General questions (p = 0.67). There were no differences in performance in the overall and subcategories among the three LLMs.

Conclusions: Our findings demonstrated satisfactory performance for all three LLMs in the MRCS Part A examination, with GPT 4.0 the only LLM that achieved the pass mark set.

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
大语言模型在英国皇家外科学院校际会员考试中的表现。
简介大型语言模型(LLM),如 Chat Generative Pre-trained Transformer(ChatGPT)和 Bard,利用在大量文本和代码数据集上经过训练的深度学习算法来生成类似人类的反应。一些研究表明,人工智能在研究生考试(包括美国医学执照考试)中的表现令人满意。我们的目标是评估人工智能在英国皇家外科学院院士(MRCS)校际考试 A 部分中的表现:我们使用 Pastest(考生常用题库)提供的 MRCS 模拟考试来评估三种 LLM 的表现:GPT-3.5、GPT 4.0 和 Bard。三百道模拟试题分别输入这三套语文能力测试卷,语文能力测试卷的答题情况被记录下来并进行分析。合格分数定为 70%:GPT-3.5、GPT 4.0 和 Bard 的总体准确率分别为 67.33%、71.67% 和 65.67%(p = 0.27)。在应用基础科学领域,GPT-3.5、GPT 4.0 和 Bard 的准确率分别为 68.89%、72.78% 和 63.33% (p = 0.15)。此外,三位法学硕士在 "外科学原理综合题 "中的正确率分别为 65.00%、70.00% 和 69.17%(p = 0.67)。三位法律硕士在总分和分项中的表现没有差异:我们的研究结果表明,三位法学硕士在 MRCS A 部分考试中的表现都令人满意,其中 GPT 4.0 是唯一达到及格分数线的法学硕士。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
CiteScore
2.40
自引率
0.00%
发文量
316
期刊介绍: The Annals of The Royal College of Surgeons of England is the official scholarly research journal of the Royal College of Surgeons and is published eight times a year in January, February, March, April, May, July, September and November. The main aim of the journal is to publish high-quality, peer-reviewed papers that relate to all branches of surgery. The Annals also includes letters and comments, a regular technical section, controversial topics, CORESS feedback and book reviews. The editorial board is composed of experts from all the surgical specialties.
期刊最新文献
Impact of endoscopic laser cricopharyngeal myotomy on lower oesophageal sphincter physiology. Kommerell's diverticulum: an unusual cause of unilateral vocal cord palsy? The novel use of a vacuum-assisted closure dressing in the management of Fournier's gangrene. Quality assessment of online patient information on upper gastrointestinal endoscopy using the modified Ensuring Quality Information for Patients tool. A new setup for single surgeon paediatric supracondylar fracture pinning.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1