Distributional Determinantal Point Process for Repulsive Clustering of Distributions

2026-07-23

AI Summary

This study addresses the problem of clustering probability distributions with repulsion to uncover semantically clear and well-separated data structures. To this end, the authors propose the distributed Determinantal Point Process (dDPP), which extends Determinantal Point Processes to the space of probability distributions for the first time. Treating distributions as atomic elements, they construct an L-ensemble using a sliced Wasserstein kernel and embed it within a generalized Bayesian mixture model to induce random partitions. The method innovatively incorporates a repulsive mechanism among distributions and introduces a utility function based on hierarchical optimal transport for posterior summarization. Experiments on single-cell gene expression and human epilepsy datasets demonstrate that the approach effectively reveals intrinsic structures, yielding highly separable and interpretable clustering results.

Abstract

We introduce the distributional determinantal point process (dDPP) as a novel repulsive point process whose atoms are probability distributions rather than points in a real space. The dDPP is constructed via an L-ensemble with a sliced Wasserstein (SW) kernel between distributions. We show its validity as a well-defined point process. In the discrete setting, we derive concentration results for plug-in estimators of the L-ensemble, the correlation kernel, and their determinants given i.i.d. samples from the distributional atoms. Leveraging this framework, we propose a distribution-valued random partition model by way of a repulsive generalized Bayesian mixture model. The model places a dDPP prior over the atoms of the mixing measure and defines a generalized likelihood based on SW distance. To summarize posterior inference, we develop a decision-theoretic approach to report a point estimate of the mixing measure as a Bayes rule under a hierarchical optimal transport utility function. The latter is a natural choice given that the mixing measure is itself a distribution over distributions. We use the proposed framework for inference with single-cell gene expression data and human epilepsy data, producing interpretable and well-separated clusters that reflect meaningful structure in the data.

Problem

• repulsive clustering
• probability distributions
• determinantal point process
• sliced Wasserstein distance
• distribution-valued clustering

Innovation

• distributional DPP
• sliced Wasserstein kernel
• repulsive clustering
• Bayesian mixture model
• optimal transport