ChatGPT and Factual Knowledge Questions Regarding Clinical Pharmacy: Response to Letter to the Editor

Abstract

Dear Editor,

The discourse surrounding the article titled “Performance of ChatGPT on Factual Knowledge Questions Regarding Clinical Pharmacy” warrants further examination and critique. The study undertook an evaluation of ChatGPT's efficacy in responding to factual knowledge questions concerning clinical pharmacy. Through a series of 264 questions, ChatGPT's responses were analyzed for accuracy, consistency, quality of the substantiation, and reproducibility, yielding notable results. ChatGPT demonstrated a 79% correctness rate, surpassing the 66% accuracy rate of pharmacists.

Acknowledging the limitations outlined in the discussion section, it is important to note that this study solely focused on factual knowledge questions. The primary objective was to determine ChatGPT's performance in responding to factual knowledge questions rather than its proficiency in clinical reasoning. Consequently, the study refrained from drawing conclusions regarding ChatGPT's impact on clinical decision-making, as this aspect falls under the scope of separate research endeavors.¹

Addressing the limitations, we argue that the scale of 264 questions, and a lack of variety are limitations of this study. The number of questions aligns with similar studies such as the USMLE Step 1, comprising 280 questions,² and the Taiwanese pharmacist licensing examination, consisting of 431 questions.³ Additionally, the span of topics covered in our questions is deemed representative of a pharmacist's factual knowledge base within clinical pharmacy.

The authors acknowledge the need for further investigation into ChatGPT's clinical applicability, for example, with longitudinal studies. Furthermore, exploring ChatGPT's capacity to provide justifications and explanations for its responses could augment its efficacy in aiding pharmacist decision-making processes. Continuous refinement and augmentation of ChatGPT are essential to strengthen its functionality as a tool for pharmacists in the clinic. Still, the indispensable expertise and interpretive skills of clinical pharmacists is pivotal to applying this information in the clinic. The factual information produced by ChatGPT holds potential as a valuable resource, however, it is imperative that the responses undergo rigorous assessment for accuracy and clinical applicability under the scrutiny of clinical pharmacists.

Sincerely,

Merel van Nuland