EHR-based prediction modelling meets multimodal deep learning: A systematic review of structured and textual data fusion methods

Electronic Health Records (EHRs) have transformed healthcare by digitally consolidating patient medical history, encompassing structured data (e.g., demographic data, lab results), and unstructured textual data (e.g., clinical notes). These data hold significant potential for predictive modelling, and recent studies have dedicated efforts to leverage the different modalities in a cohesive and effective manner to improve predictive accuracy. This Systematic Literature Review (SLR) addresses the application of Multimodal Deep Learning (MDL) methods in EHR-based prediction modelling, specifically through the fusion of structured and textual data. Following PRISMA guidelines, we conducted a comprehensive literature search across six article databases, using a carefully designed search string. After applying inclusion and exclusion criteria, we selected 77 primary studies. Data extraction was standardized using a structured form based on the CHARMS checklist. We categorized and analysed the fusion strategies employed across the studies. By combining structured and textual data at the input level, early fusion enabled models to learn joint feature representations from the beginning, whether in vectorized representations or data textualization. Intermediate fusion, which delays integration, was particularly useful for tasks where each modality provides unique insights that need to be processed independently before being combined. Late fusion enabled modularity by integrating outputs from unimodal models, which is suitable when EHR structured and textual data have varying quality or reliability. We also identified trends and open issues that need attention. This review contributes a comprehensive understanding of EHR data fusion practices using MDL, highlighting potential pathways for future research and development in health informatics.