Although large language models (LLMs) have the potential to transform biomedical research, their ability to reason accurately across complex, data-rich domains remains unproven. To address this research gap, we introduce CARDBiomedBench, a large-scale question-and-answer benchmark for evaluating LLMs in biomedical science. This pilot release focuses on neurodegenerative disease research, a field requiring the integration of genomics, pharmacology, and statistical reasoning. CARDBiomedBench includes more than 68 000 curated question-answer pairs generated through expert annotation and structured data augmentation. The questions spanned ten biological categories and nine reasoning types, based on publicly available resources, such as genome-wide association studies, summary data-based mendelian randomisation results, and regulatory drug databases. We assessed model responses using BioScore, a rubric-based evaluation system that measures response accuracy (response quality rate, RQR) and the ability to abstain from incorrect answers (safety rate). Testing 18 state-of-the-art LLMs revealed considerable gaps. Claude-3.5-Sonnet achieved high caution but low accuracy (safety rate 75%, RQR 24%), whereas GPT-4.1 showed the opposite trade-off (safety rate 7%, RQR 51%). No model showed a successful balance of both metrics. CARDBiomedBench provides a new standard for benchmarking biomedical LLMs, revealing key limitations in existing models and offering a scalable path towards safer, more effective artificial intelligence systems in scientific research.
Developments in large language models (LLMs) in the past 2 years have shifted the focus from text, image, and audio generation to LLMs capable of multistep reasoning (thinking). The development of LLMs is particularly important for medicine and health care, but the translation of these models has been limited by the black-box nature of previous LLMs. New reasoning-driven LLMs incorporate chain-of-thought prompting and reveal intermediate reasoning steps, offering transparency and traceability, potentially improving the clinical adoption and utility of LLMs. In this Viewpoint, we examine four emerging reasoning-driven LLMs, namely OpenAI's o1 and o3-mini, Google's Gemini 2.0 Flash Thinking, and DeepSeek R1. We compare their methodological approaches, benchmark their performance on medical question-answering tasks, and assess their potential for clinical integration. We highlight both opportunities and challenges associated with deploying reasoning-driven LLMs. Key future considerations include real-world validation, rigorous benchmarking with ethical safeguards, and advancements in improving the efficiency and sustainability of reasoning-driven LLMs. Addressing these challenges will enable the fine-tuning of these LLMs for specific medical applications, enhancing their potential clinical decision support, patient education, medical training, and evidence synthesis.
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：