Model-agnostic confidence measurement for aggregating multimodal ensemble models in automatic diagnostic systems

Abstract

Automatic diagnostic systems (ADSs) have been garnering increased attention because they can alleviate the workload of clinicians by assisting in diagnosis and offering low-cost access to healthcare for people in medically underserved areas. ADS can suggest potential diseases by analyzing a patient's self-report. Previous research on ADS has leveraged diagnostic case data from various patients and medical knowledge to diagnose diseases, with multimodal ensemble methods proving particularly effective. However, the existing multimodal ensemble method combines the probabilities of different models in the aggregating process, which can not properly combine the probabilities that are produced by different criteria. To address these issues, we propose an effective aggregation framework for multimodal ensembles that can properly aggregate model-agnostic confidence scores and predictions from each model. Our framework transforms probability scores from different criteria into unified aggregation rule-based scores and reflects the gap between the probabilities that may be blurred in the aggregation process through the confidence score. In particular, The proposed confidence measurement method employs a post-analysis approach with the developed model or algorithm, making it adaptable in a model-agnostic manner and suitable for multimodal ensemble learning that utilizes heterogeneous prediction results. Our experimental results demonstrate that our framework outperforms existing approaches by more effectively leveraging the strengths of each ensemble member.