Mixtures of Heterogeneous Experts

Abstract

No single machine learning algorithm is most accurate for all problems due to the effect of an algorithm's inductive bias. Research has shown that a combination of experts of the same type, referred to as a mixture of homogeneous experts, can increase the accuracy of ensembles by reducing the adverse effect of an algorithm's inductive bias. However, the predictive power of a mixture of homogeneous experts is still limited by the inductive bias of the algorithm that makes up the mixture. For this reason, combinations of different machine learning algorithms, referred to as a mixture of heterogeneous experts, has been proposed to take advantage of the strengths of different machine learning algorithms and to reduce the adverse effects of the inductive biases of these algorithms. This paper presents a mixture of heterogeneous experts, and evaluates its performance to that of a number of mixtures of homogeneous experts on a set of classification problems. The results indicate that a mixture of heterogeneous experts aggregates the advantages of experts, increasing the accuracy of predictions. The mixture of heterogeneous experts not only outperformed all homogeneous ensembles on two of the datasets, but also achieved the best overall accuracy rank across the various datasets.