Highly improve the accuracy of clustering algorithms based on shortest path distance

Abstract

Dataset-amelioration methods improve clustering accuracy by introducing gravitation between neighboring objects, pulling them closer together. However, in overlapping datasets, the gravitation can also pull adjacent clusters closer, which will degrade data distribution. Highly Improving the Accuracy of Clustering (HIAC) constructs a probability curve to select a global threshold that eliminates inter-cluster gravitation, thereby aggregating objects within the same cluster. Nonetheless, the global threshold may erroneously retain inter-cluster gravitation while removing intra-cluster gravitation, potentially resulting in the formation of new tiny clusters and the deviation of boundary objects. To address this issue, we propose the HIACSP algorithm (Highly Improving the Accuracy of Clustering Algorithms based on Shortest Path Distance). HIACSP defines the weight of the shortest path between objects as a novel distance metric, denoted as ${\delta }_{SP}$. This new metric prompts the K nearest neighbors identified by ${\delta }_{SP}$ to be biased toward the cluster core and belong to the same cluster. Consequently, only intra-cluster gravitation forces are retained without relying on the threshold, thus preventing the formation of tiny clusters and the deviation of boundary objects. Additionally, based on SP-KNN, the boundary score is devised to identify actual boundary objects. By pulling boundary objects toward the cluster core using the gravitation acting on them by SP-KNN, overlapping clusters can be well-separated, and no clusters will be over-divided. Extensive experiments have been conducted to validate HIACSP. The experimental results show that HIACSP achieves an average improvement in clustering accuracy of 19.9% (Adjusted Rand Index, ARI), 14.8% (Normalized Mutual Information, NMI), 12.0% (Fowlkes-Mallows Index, FMI), 11.0% (Purity, PUR), and 14.8% (V-Measure, VM) across five evaluation metrics, outperforming baseline algorithms by at least 5.7% (ARI), 3.9% (NMI), 3.2% (FMI), 3.6% (PUR), and 3.9% (VM). The code and datasets are available at https://github.com/XJaiYH/HIACSP.