Research on clustering algorithm based on spark

Abstract

With the rapid development of sensors and positioning technology, a huge amount of GPS data generates every day and night. Taking cabs as an example, behind the GPS track information of cabs, there is a large amount of information to be mined, which is crucial for urban governance and consumer behavior analysis. In this paper, we will analyze point data of cab with clustering algorithm, optimize K-means by utilizing the Canopy algorithm for pre-clustering, and parallelize the implementation of the algorithm based on the spark framework. Experiments show that the improved clustering algorithm works well, and the computational efficiency and speedup also improve effectively.