A deflation-adjusted Bayesian information criterion for selecting the number of clusters in K-means clustering

Abstract

A deflation-adjusted Bayesian information criterion is proposed by introducing a closed-form adjustment to the variance estimate after K-means clustering. An expected lower bound of the deflation in the variance estimate after K-means clustering is derived and used as an adjustment factor for the variance estimates. The deflation-adjusted variance estimates are applied to the Bayesian information criterion under the Gaussian model for selecting the number of clusters. The closed-form expression makes the proposed deflation-adjusted Bayesian information criterion computationally efficient. Simulation studies show that the deflation-adjusted Bayesian information criterion performs better than other existing clustering methods in some situations, including K-means clustering with the number of clusters selected by standard Bayesian information criteria, the gap statistic, the average silhouette score, the prediction strength, and clustering using a Gaussian mixture model with the Bayesian information criterion. The proposed method is illustrated through a real data application for clustering human genomic data from the 1000 Genomes Project.