An Approximative Bayes-Optimal Kernel Classifier Based on Version Space Reduction

Abstract

The Bayes-optimal classifier is defined as a classifier that induces an hypothesis able to minimize the prediction error for any given sample in binary classification problems. Finding the Bayes-optimal classifier is an intractable problem. It is known that it is approximately equivalent to the center of mass of the version space, which is given by the set of all classifiers consistent with the training set. Previously solutions to find the center of mass are not feasible, as they present a high computational cost, and do not work properly in non-linear separable problems. Aiming to solve these problems, this paper presents the Dual Version Space Reduction Machine (Dual VSRM), an effective kernel method to approximate the center of mass of the version space. The Dual VSRM algorithm employs successive reductions of the version space based on an oracle's decision. As an oracle, we propose the Ensemble of Dissimilar Balanced Kernel Perceptrons (EBPK). EBPK enhances the accuracy of each individual classifier by balancing the final hyperplane solution while maximizing the diversity of its components by applying a dissimilarity measure. In order to evaluate the proposed methods, we conduct an experimental evaluation on 7 datasets. We compare the performance of our proposed methods against several baselines. Our results for EBKP indicate the strategies for improving individual accuracy and diversity of the ensemble components work properly. Also, the Dual VSRM consistently outperforms the baselines, indicating that the proposed method generates a better approximation to the center of mass.