Information cut and information forces for clustering

Abstract

We define an information-theoretic divergence measure between probability density functions (pdfs) that has a deep connection to the cut in graph-theory. This connection is revealed when the pdfs are estimated by the Parzen method with a Gaussian kernel. We refer to our divergence measure as the information cut. The information cut provides us with a theoretically sound criterion for cluster evaluation. In this paper we show that it can be used to merge clusters. The initial clusters are obtained based on the related concept of information forces. We create directed trees by selecting the predecessor of a node (pattern) according to the direction of the information force acting on the pattern. Each directed tree corresponds to a cluster, hence enabling us to obtain an initial partitioning of the data set. Subsequently, we utilize the information cut as a cluster evaluation function to merge clusters until the predefined number of clusters is reached. We demonstrate the performance of our novel information-theoretic clustering method when applied to both artificially created data and real data, with encouraging results.