Automatic medical report generation based on deep learning: A state of the art survey

Abstract

With the increasing popularity of medical imaging and its expanding applications, posing significant challenges for radiologists. Radiologists need to spend substantial time and effort to review images and manually writing reports every day. To address these challenges and speed up the process of patient care, researchers have employed deep learning methods to automatically generate medical reports. In recent years, researchers have been increasingly focusing on this task and a large amount of related work has emerged. Although there have been some review articles summarizing the state of the art in this field, their discussions remain relatively limited. Therefore, this paper provides a comprehensive review of the latest advancements in automatic medical report generation, focusing on four key aspects: (1) describing the problem of automatic medical report generation, (2) introducing datasets of different modalities, (3) thoroughly analyzing existing evaluation metrics, (4) classifying existing studies into five categories: retrieval-based, domain knowledge-based, attention-based, reinforcement learning-based, large language models-based, and merged model. In addition, we point out the problems in this field and discuss the directions of future challenges. We hope that this review provides a thorough understanding of automatic medical report generation and encourages the continued development in this area.