Combining classification improvements by ensemble processing

Abstract

The k-nearest neighbor (KNN) classification is a simple and effective classification approach. However, improving performance of the classifier is still attractive. Combining multiple classifiers is an effective technique for improving accuracy. There are many general combining algorithms, such as Bagging, Boosting, or Error Correcting Output Coding that significantly improve the classifier such as decision trees, rule learners, or neural networks. Unfortunately, these combining methods developed do not improve the nearest neighbor classifiers. In this paper, first, we present a new approach to combine multiple KNN classifiers based on different distance functions, in which we apply multiple distance functions to improve the performance of the k-nearest neighbor classifier. Second, we develop a combining method, in which the weights of the distance function are learnt by genetic algorithm. Finally, combining classifiers in error correcting output coding, are discussed. The proposed algorithms seek to increase generalization accuracy when compared to the basic k-nearest neighbor algorithm. Experiments have been conducted on some benchmark datasets from the UCI machine learning repository. The results show that the proposed algorithms improve the performance of the k-nearest neighbor classification.