MedicalGLM: A Pediatric Medical Question Answering Model with a quality evaluation mechanism

Abstract

Objective:

Large Language models (LLMs) have a wide range of medical applications, especially in scenarios such as question-answering. However, existing models face the challenge of accurately assessing the quality of information when generating medical information, which may lead to the inability to effectively distinguish beneficial and harmful information, thus affecting the quality of question-answering. This study aims to improve the information quality and practicability of medical question-answering.

Methods:

This study proposes MedicalGLM, a fine-tuning model based on a quality evaluation mechanism. Specifically, MedicalGLM contains a reward model for assessing the quality of medical QA. It adjusts its training process by returning the assessment scores to the QA model as penalties through a quality score loss function.

Results:

The experimental results indicate that MedicalGLM achieved the highest scores among the evaluated models in the Rouge-1, Rouge-2, Rouge-L, and BLEU metrics, with values of 54.90, 28.02, 44.50, and 32.61, respectively. Its proficiency in generating responses for the pediatric medical quiz task is notably superior to other prevailing LLMs in the medical domain.

Conclusion:

MedicalGLM significantly improves the quality and practicability of the generated information of the medical question-answering model by introducing a quality evaluation mechanism, which provides an effective improvement idea for researching medical large language models. Our code and model are publicly available for further research on https://github.com/wangxinwwang/MedicalGLM.