Towards a robust multi-view information bottleneck using Cauchy–Schwarz divergence

Abstract

Efficiently preserving task-relevant information while removing noise and redundancy in multi-view data remains a core challenge. The information bottleneck principle offers an information-theoretic framework to compress data while retaining essential information for the task. However, estimating mutual information in high-dimensional spaces is computationally intractable. Commonly used variational methods introduce uncertainty and risk performance degradation. To overcome these limitations, we propose a robust deterministic multi-view information bottleneck framework that circumvents the need for variational inference or distributional assumptions. Specifically, we present a non-parametric mutual information estimation based on the Cauchy–Schwarz divergence, eliminating the need for auxiliary neural estimators and significantly simplifying the optimization of the information bottleneck. Leveraging this mutual information measure, we design a neural network framework that robustly compresses high-dimensional multi-view data into a low-dimensional representation, extracting task-relevant features that adhere to both sufficiency and minimality. Additionally, attention mechanisms are employed to fuse compact features across different views, capturing interdependencies and enhancing the integration of complementary information. This fusion process improves the robustness of the overall representation. Statistical analysis using the Nemenyi test shows statistically significant differences in performance between our method and existing algorithms, with a critical distance (CD = 1.856, p-value <0.05), demonstrating the superiority of our approach. Experimental results on synthetic data highlight the framework’s robustness in handling noise and redundancy, demonstrating its effectiveness in challenging environments. Validation on eight real-world datasets, including electroencephalography and Alzheimer’s neuroimaging data, confirms its superior performance, particularly with limited training samples. The implementation is available at https://github.com/archy666/CSMVIB.