Is ChatGPT reliable and accurate in answering pharmacotherapy-related inquiries in both Turkish and English?

IF 1.3 Q3 EDUCATION, SCIENTIFIC DISCIPLINES Currents in Pharmacy Teaching and Learning Pub Date : 2024-05-03 DOI:10.1016/j.cptl.2024.04.017

Nur Ozturk , Irem Yakak , Melih Buğra Ağ , Nilay Aksoy

{"title":"Is ChatGPT reliable and accurate in answering pharmacotherapy-related inquiries in both Turkish and English?","authors":"Nur Ozturk , Irem Yakak , Melih Buğra Ağ , Nilay Aksoy","doi":"10.1016/j.cptl.2024.04.017","DOIUrl":null,"url":null,"abstract":"<div><h3>Introduction</h3><p>Artificial intelligence (AI), particularly ChatGPT, is becoming more and more prevalent in the healthcare field for tasks such as disease diagnosis and medical record analysis. The objective of this study is to evaluate the proficiency and accuracy of ChatGPT in different domains of clinical pharmacy cases and queries.</p></div><div><h3>Methods</h3><p>The study NAPLEX® Review Questions, 4th edition, pertaining to 10 different chronic conditions compared ChatGPT's responses to pharmacotherapy cases and questions obtained from McGraw Hill's, alongside the answers provided by the book's authors. The proportion of correct responses was collected and analyzed using the Statistical Package for the Social Sciences (SPSS) version 29.</p></div><div><h3>Results</h3><p>When tested in English, ChatGPT had substantially higher mean scores than when tested in Turkish. The average accurate score for English and Turkish was 0.41 ± 0.49 and 0.32 ± 0.46, respectively, <em>p</em> = 0.18. Responses to queries beginning with “Which of the following is correct?” are considerably more precise than those beginning with “Mark all the incorrect answers?” 0.66 ± 0.47 as opposed to 0.16 ± 0.36; <em>p</em> = 0.01 in English language and 0.50 ± 0.50 as opposed to 0.14 ± 0.34; <em>p</em> < 0.05in Turkish language.</p></div><div><h3>Conclusion</h3><p>ChatGPT displayed a moderate level of accuracy while responding to English inquiries, but it displayed a slight level of accuracy when responding to Turkish inquiries, contingent upon the question format. Improving the accuracy of ChatGPT in languages other than English requires the incorporation of several components. The integration of the English version of ChatGPT into clinical practice has the potential to improve the effectiveness, precision, and standard of patient care provision by supplementing personal expertise and professional judgment. However, it is crucial to utilize technology as an adjunct and not a replacement for human decision-making and critical thinking.</p></div>","PeriodicalId":47501,"journal":{"name":"Currents in Pharmacy Teaching and Learning","volume":"16 7","pages":"Article 102101"},"PeriodicalIF":1.3000,"publicationDate":"2024-05-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Currents in Pharmacy Teaching and Learning","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1877129724001205","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Introduction

Artificial intelligence (AI), particularly ChatGPT, is becoming more and more prevalent in the healthcare field for tasks such as disease diagnosis and medical record analysis. The objective of this study is to evaluate the proficiency and accuracy of ChatGPT in different domains of clinical pharmacy cases and queries.

Methods

The study NAPLEX® Review Questions, 4th edition, pertaining to 10 different chronic conditions compared ChatGPT's responses to pharmacotherapy cases and questions obtained from McGraw Hill's, alongside the answers provided by the book's authors. The proportion of correct responses was collected and analyzed using the Statistical Package for the Social Sciences (SPSS) version 29.

Results

When tested in English, ChatGPT had substantially higher mean scores than when tested in Turkish. The average accurate score for English and Turkish was 0.41 ± 0.49 and 0.32 ± 0.46, respectively, p = 0.18. Responses to queries beginning with “Which of the following is correct?” are considerably more precise than those beginning with “Mark all the incorrect answers?” 0.66 ± 0.47 as opposed to 0.16 ± 0.36; p = 0.01 in English language and 0.50 ± 0.50 as opposed to 0.14 ± 0.34; p < 0.05in Turkish language.

Conclusion

ChatGPT displayed a moderate level of accuracy while responding to English inquiries, but it displayed a slight level of accuracy when responding to Turkish inquiries, contingent upon the question format. Improving the accuracy of ChatGPT in languages other than English requires the incorporation of several components. The integration of the English version of ChatGPT into clinical practice has the potential to improve the effectiveness, precision, and standard of patient care provision by supplementing personal expertise and professional judgment. However, it is crucial to utilize technology as an adjunct and not a replacement for human decision-making and critical thinking.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

在用土耳其语和英语回答与药物治疗相关的咨询时，ChatGPT 是否可靠、准确？

引言人工智能（AI），尤其是 ChatGPT，在医疗保健领域正变得越来越普遍，用于疾病诊断和病历分析等任务。本研究的目的是评估 ChatGPT 在不同领域的临床药学案例和查询中的熟练程度和准确性：研究 NAPLEX® 第 4 版复习题，涉及 10 种不同的慢性疾病，比较了 ChatGPT 对从 McGraw Hill 获得的药物治疗案例和问题的回答，以及该书作者提供的答案。使用社会科学统计软件包（SPSS）29 版收集和分析了正确回答的比例：用英语测试时，ChatGPT 的平均得分大大高于用土耳其语测试时的平均得分。英语和土耳其语的平均正确率分别为 0.41 ± 0.49 和 0.32 ± 0.46，p = 0.18。对以 "以下哪项是正确的？"开头的询问的回答要比以 "标出所有错误答案？"开头的回答精确得多。英语为 0.66 ± 0.47，而英语为 0.16 ± 0.36; p = 0.01；英语为 0.50 ± 0.50，而英语为 0.14 ± 0.34; p 结论：ChatGPT 在回答英语问询时显示出中等水平的准确性，但在回答土耳其语问询时，根据问题格式的不同，显示出轻微的准确性。要提高 ChatGPT 在英语以外语言中的准确性，需要整合多个组件。将英语版 ChatGPT 融入临床实践有可能通过补充个人专业知识和专业判断来提高病人护理的有效性、准确性和标准。不过，关键是要将技术作为辅助手段，而不是替代人类决策和批判性思维。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊