An improved measure for data clustering in high dimensional space

Abstract

The k-means clustering fails to correctly cluster the data points in high dimensional space, primarily for employing Euclidean norm as the distance metric. The Euclidean metric increases with the increase in data dimension, thus posing difficulty to segregate intra-cluster and inter-cluster data points. Adoption of k-means clustering, realized with Euclidean distance norm, often misguides the selection of cluster centres in a given iteration. This paper proposes a novel approach to k-means clustering algorithm by replacing the Euclidean distance metric by a new one. The merit of the proposed metric lies in keeping the distance low, even for large dimensional data points. The new metric enables the algorithm to correctly select the cluster centres over the iterations. Experiments undertaken revealed that the said distance metric based k-means clustering outperforms the traditional one by a large margin.