Co-Occurrence Relationship Driven Hierarchical Attention Network for Brain CT Report Generation

Abstract

Automatic generation of medical reports for Brain Computed Tomography (CT) imaging is crucial for helping radiologists make more accurate clinical diagnoses efficiently. Brain CT imaging typically contains rich pathological information, including common pathologies that often co-occur in one report and rare pathologies that appear in medical reports with lower frequency. However, current research ignores the potential co-occurrence between common pathologies and pays insufficient attention to rare pathologies, severely restricting the accuracy and diversity of the generated medical reports. In this paper, we propose a Co-occurrence Relationship Driven Hierarchical Attention Network (CRHAN) to improve Brain CT report generation by mining common and rare pathologies in Brain CT imaging. Specifically, the proposed CRHAN follows a general encoder-decoder framework with two novel attention modules. In the encoder, a co-occurrence relationship guided semantic attention (CRSA) module is proposed to extract the critical semantic features by embedding the co-occurrence relationship of common pathologies into semantic attention. In the decoder, a common-rare topic driven visual attention (CRVA) module is proposed to fuse the common and rare semantic features as sentence topic vectors, and then guide the visual attention to capture important lesion features for medical report generation. Experiments on the Brain CT dataset demonstrate the effectiveness of the proposed method.