🤖 AI Summary
This work addresses the limitations of existing cluster validity indices in unsupervised clustering, where the absence of ground-truth labels and the prevalence of non-convex, irregular, or unevenly distributed data structures hinder reliable evaluation. To overcome these challenges, the authors propose the Central Description Length (CDL) metric, an information-theoretic approach that jointly captures intra-cluster compactness and centroid displacement. CDL estimates an upper bound on the description length of cluster centers, enabling label-free assessment of arbitrary clustering outcomes without reliance on Euclidean distance or kernel functions, thus accommodating clusters of any shape. Empirical results demonstrate that CDL more accurately identifies the true number of clusters and achieves higher Adjusted Rand Index (ARI) scores on synthetic non-convex datasets. Moreover, when applied to embedded representations of MNIST, CIFAR-10, and STL-10, CDL robustly estimates cluster counts closely aligned with the true number of classes.
📝 Abstract
Selecting a clustering algorithm and its hyperparameters without labels is a common difficulty in engineering machine learning pipelines that work with unsupervised analysis of sensor, image, or process data. Clustering validation indices (CVIs) provide internal scores for ranking candidate clusterings, but most popular CVIs are built from Euclidean compactness and separation terms and so tend to favour compact, convex partitions. Their performance is known to degrade on non convex, irregular, or variable density data, where kernel transformations or alternative distance measures are typically used at the cost of additional tuning and computation. This paper introduces the Central Description Length (CDL) clustering validation index. CDL uses the observed within cluster compactness, the estimated cluster centers, and the estimated cluster covariances to compute a probabilistic upper bound on the description length associated with the unobservable true cluster centers. The bound condenses intra cluster compactness and centroid displacement into a single computable quantity and is evaluated on the partition produced by any clustering algorithm. The implementation uses only observable quantities (the data, the partition, the estimated centers, and the estimated covariances) and does not use ground truth labels. On synthetic benchmarks with non convex and arbitrary shape clusters, CDL-CVI selected the reference number of clusters more often and reached higher Adjusted Rand Index (ARI) values than the conventional CVIs we tested, without an additional kernel preprocessing stage. On image benchmarks (MNIST, CIFAR-10, STL-10) clustered from frozen unsupervised embeddings, CDL-CVI returned cluster numbers close to the reference class counts across K-means, DBSCAN, and spectral clustering in the reported trials.