Research and improve on K-means algorithm based on hadoop

Abstract

With the advent of the big data era, traditional data mining algorithm becomes incompetent for the task of massive data analysis, management and mining. The development of cloud computing brings new life to algorithm parallelization. In this paper, we have studied the K-means algorithm, one of the clustering algorithm. Then we attempt to improves this algorithm via the method that sample the large-scale data and use convex hull and opposite Chung points to solve the initial two cluster centers. We also take the MapReduce programming model to parallelize the whole process. Finally, using the Reuters news set 21578 as a data source, comparative experiments under different distance measure, serial to parallel, and different cluster nodes have been done to verify the efficiency of the improved algorithm. Results show that compared with serial algorithm, the improved parallel algorithm improves obviously both in reliability and efficiency with the increase of cluster nodes and data size.