From failure to fusion: A survey on learning from bad machine learning models

Abstract

Machine learning (ML) models are ubiquitous across diverse applications; however, only a fraction achieves optimal performance, often leading to the deployment of a singular model while dismissing others as experimental failures. This paper challenges this commonly accepted practice by systematically investigating the utility of suboptimal ML models. We posit that these models encapsulate valuable information regarding data biases, architectural limitations, and systemic misalignments, which can be leveraged to enhance overall system performance. Central to our approach is the integration of information fusion techniques, which combine heterogeneous data sources to robustly analyze and contextualize the errors and biases present in underperforming models. Our methodology includes advanced negative knowledge distillation, as well as error-based curriculum learning frameworks that are derived from multiple data modalities. We propose a comprehensive debugging framework that utilizes meta-learning for failure detection and correction to enable continuous improvement through rigorous cross-validation and iterative refinement. This study stresses the importance of documenting negative outcomes to promote transparency and foster interdisciplinary collaboration to build resilient and generalizable ML systems, particularly in information fusion. We advocate for a paradigm shift in the ML community and urge both researchers and institutions to systematically harness the insights derived from so-called "failed" models. We then conclude this paper by discussing several challenges and possible pathways for future research.