Performance of Novel GPT-4 in Otolaryngology Knowledge Assessment.

IF 0.6 Q4 SURGERY Indian Journal of Otolaryngology and Head and Neck Surgery Pub Date : 2024-12-01 Epub Date: 2024-08-03 DOI:10.1007/s12070-024-04935-x

Lucy Revercomb, Aman M Patel, Daniel Fu, Andrey Filimonov

{"title":"Performance of Novel GPT-4 in Otolaryngology Knowledge Assessment.","authors":"Lucy Revercomb, Aman M Patel, Daniel Fu, Andrey Filimonov","doi":"10.1007/s12070-024-04935-x","DOIUrl":null,"url":null,"abstract":"Purpose: GPT-4, recently released by OpenAI, improves upon GPT-3.5 with increased reliability and expanded capabilities, including user-specified, customizable GPT-4 models. This study aims to investigate updates in GPT-4 performance vs. GPT-3.5 on Otolaryngology board-style questions.Methods: 150 Otolaryngology board-style questions were obtained from the BoardVitals question bank. These questions, which were previously assessed with GPT-3.5, were inputted into standard GPT-4 and a custom GPT-4 model designed to specialize in Otolaryngology board-style questions, emphasize precision, and provide evidence-based explanations.Results: Standard GPT-4 correctly answered 72.0% and custom GPT-4 correctly answered 81.3% of the questions, vs. GPT-3.5 which answered 51.3% of the same questions correctly. On multivariable analysis, custom GPT-4 had higher odds of correctly answering questions than standard GPT-4 (adjusted odds ratio 2.19, P = 0.015). Both GPT-4 and custom GPT-4 demonstrated a decrease in performance between questions rated as easy and hard (P < 0.001).Conclusions: Our study suggests that GPT-4 has higher accuracy than GPT-3.5 in answering Otolaryngology board-style questions. Our custom GPT-4 model demonstrated higher accuracy than standard GPT-4, potentially as a result of its instructions to specialize in Otolaryngology board-style questions, select exactly one answer, and emphasize precision. This demonstrates custom models may further enhance utilization of ChatGPT in medical education.","PeriodicalId":49190,"journal":{"name":"Indian Journal of Otolaryngology and Head and Neck Surgery","volume":"76 6","pages":"6112-6114"},"PeriodicalIF":0.6000,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11569072/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Indian Journal of Otolaryngology and Head and Neck Surgery","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1007/s12070-024-04935-x","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/8/3 0:00:00","PubModel":"Epub","JCR":"Q4","JCRName":"SURGERY","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: GPT-4, recently released by OpenAI, improves upon GPT-3.5 with increased reliability and expanded capabilities, including user-specified, customizable GPT-4 models. This study aims to investigate updates in GPT-4 performance vs. GPT-3.5 on Otolaryngology board-style questions.

Methods: 150 Otolaryngology board-style questions were obtained from the BoardVitals question bank. These questions, which were previously assessed with GPT-3.5, were inputted into standard GPT-4 and a custom GPT-4 model designed to specialize in Otolaryngology board-style questions, emphasize precision, and provide evidence-based explanations.

Results: Standard GPT-4 correctly answered 72.0% and custom GPT-4 correctly answered 81.3% of the questions, vs. GPT-3.5 which answered 51.3% of the same questions correctly. On multivariable analysis, custom GPT-4 had higher odds of correctly answering questions than standard GPT-4 (adjusted odds ratio 2.19, P = 0.015). Both GPT-4 and custom GPT-4 demonstrated a decrease in performance between questions rated as easy and hard (P < 0.001).

Conclusions: Our study suggests that GPT-4 has higher accuracy than GPT-3.5 in answering Otolaryngology board-style questions. Our custom GPT-4 model demonstrated higher accuracy than standard GPT-4, potentially as a result of its instructions to specialize in Otolaryngology board-style questions, select exactly one answer, and emphasize precision. This demonstrates custom models may further enhance utilization of ChatGPT in medical education.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

新版 GPT-4 在耳鼻喉科知识评估中的表现。

目的：OpenAI 最近发布的 GPT-4 在 GPT-3.5 的基础上进行了改进，提高了可靠性并扩展了功能，包括用户指定、可定制的 GPT-4 模型。本研究旨在调查 GPT-4 与 GPT-3.5 在耳鼻喉科板式试题上的性能对比。这些问题以前曾用 GPT-3.5 进行过评估，现在被输入到标准 GPT-4 和定制的 GPT-4 模型中，该模型专为耳鼻喉科委员会风格的问题而设计，强调精确性并提供基于证据的解释：标准 GPT-4 正确回答了 72.0% 的问题，自定义 GPT-4 正确回答了 81.3% 的问题，而 GPT-3.5 正确回答了 51.3% 的问题。经多变量分析，定制的 GPT-4 比标准的 GPT-4 正确回答问题的几率更高（调整后的几率比 2.19，P = 0.015）。GPT-4 和自定义 GPT-4 在被评为简单和困难的问题之间的成绩都有所下降（P 结论：GPT-4 和自定义 GPT-4 在被评为简单和困难的问题之间的成绩都有所下降：我们的研究表明，与 GPT-3.5 相比，GPT-4 在回答耳鼻喉科委员会类型的问题时具有更高的准确性。我们的定制 GPT-4 模型比标准 GPT-4 显示出更高的准确性，这可能是由于其指示专门针对耳鼻喉科委员会风格的问题、准确选择一个答案以及强调精确性的结果。这表明自定义模型可进一步提高 ChatGPT 在医学教育中的应用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Indian Journal of Otolaryngology and Head and Neck Surgery Medicine-Otorhinolaryngology

CiteScore

0.80

自引率

0.00%

发文量

226

审稿时长

6-12 weeks

期刊介绍： Indian Journal of Otolaryngology and Head & Neck Surgery was founded as Indian Journal of Otolaryngology in 1949 as a scientific Journal published by the Association of Otolaryngologists of India and was later rechristened as IJOHNS to incorporate the changes and progress. IJOHNS, undoubtedly one of the oldest Journals in India, is the official publication of the Association of Otolaryngologists of India and is about to publish it is 67th Volume in 2015. The Journal published quarterly accepts articles in general Oto-Rhino-Laryngology and various subspecialities such as Otology, Rhinology, Laryngology and Phonosurgery, Neurotology, Head and Neck Surgery etc. The Journal acts as a window to showcase and project the clinical and research work done by Otolaryngologists community in India and around the world. It is a continued source of useful clinical information with peer review by eminent Otolaryngologists of repute in their respective fields. The Journal accepts articles pertaining to clinical reports, Clinical studies, Research articles in basic and applied Otolaryngology, short Communications, Clinical records reporting unusual presentations or lesions and new surgical techniques. The journal acts as a catalyst and mirrors the Indian Otolaryngologist’s active interests and pursuits. The Journal also invites articles from senior and experienced authors on interesting topics in Otolaryngology and allied sciences from all over the world. The print version is distributed free to about 4000 members of Association of Otolaryngologists of India and the e-Journal shortly going to make its appearance on the Springer Board can be accessed by all the members. Association of Otolaryngologists of India and M/s Springer India group have come together to co-publish IJOHNS from January 2007 and this bondage is going to provide an impetus to the Journal in terms of international presence and global exposure.