The generalized underlap coefficient with an application in clustering

2026-02-22

AI Summary

Existing measures of distributional discrepancy are largely confined to pairwise comparisons and struggle to capture the overall separation among multiple multivariate distributions. This work extends the underlap coefficient—originally defined for two groups—to the multivariate, multi-group setting, establishes its theoretical connection to total variation distance, and interprets it as a measure of dependence between group labels and covariates. To enable practical computation, we develop an efficient estimation algorithm based on importance sampling and flexible density estimation. Experiments on two real-world datasets demonstrate that the proposed method effectively quantifies the dependence between covariates and cluster assignments, offering a novel tool for assessing clustering validity, including testing the single-weight hypothesis.

Technology Category

• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Quantifying distributional separation across groups is fundamental in statistical learning and scientific discovery, yet most classical discrepancy measures are tailored to two-group comparisons. We generalize the underlap coefficient (UNL), a multi-group separation measure, to multivariate variables. We establish key properties of UNL and provide an explicit connection to the total variation. We further interpret the UNL as a dependence measure between a group label and variables of interest and compare it with mutual information. We propose an importance sampling estimator of the UNL that can be combined with flexible density estimators. The utility of the UNL for assessing partition-covariate dependence in clustering is highlighted in detail, where it is particularly useful for evaluating the single-weights assumption in covariate-dependent mixture models. Finally we illustrate the application of the UNL in clustering using two real world datasets.

Problem

• distributional separation
• multi-group comparison
• clustering
• covariate dependence
• underlap coefficient

Innovation

• underlap coefficient
• multivariate generalization
• distributional separation
• importance sampling estimator
• clustering evaluation