Assessment of large language models in medical quizzes for clinical chemistry and laboratory management: implications and applications for healthcare artificial intelligence.

Abstract

Large language models (LLMs) have demonstrated high performance across various fields due to their ability to understand, generate, and manipulate human language. However, their potential in specialized medical domains, such as clinical chemistry and laboratory management, remains underexplored. This study evaluated the performance of nine LLMs using zero-shot prompting on 109 clinical problem-based quizzes from peer-reviewed journal articles in the Laboratory Medicine Online (LMO) database. These quizzes covered topics in clinical chemistry, toxicology, and laboratory management. The models, including GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro, along with their earlier or smaller versions, were assigned roles as clinical chemists or laboratory managers to simulate real-world decision-making scenarios. Among the evaluated models, GPT-4o achieved the highest overall accuracy, correctly answering 81.7% of the quizzes, followed by GPT-4 Turbo (76.1%), Claude 3 Opus (74.3%), and Gemini 1.5 Pro (69.7%), while the lowest performance was observed with Gemini 1.0 Pro (51.4%). GPT-4o performed exceptionally well across all quiz types, including single-select, open-ended, and multiple-select questions, and demonstrated particular strength in quizzes involving figures, tables, or calculations. These findings highlight the ability of LLMs to effectively apply their pre-existing knowledge base to specialized clinical chemistry inquiries without additional fine-tuning. Among the evaluated models, GPT-4o exhibited superior performance across different quiz types, underscoring its potential utility in assisting healthcare professionals in clinical decision-making.