Hierarchical shot-noise Cox process mixtures for clustering across groups

2025-10-16

AI Summary

Bayesian nonparametric clustering of multiple groups of partially exchangeable data faces two key challenges: a trade-off between cluster structure flexibility and density estimation accuracy, and cross-group cluster fragmentation under large sample sizes. To address these, we propose a hierarchical shot-noise Cox process mixture model, which employs kernel functions to characterize group-specific atoms clustered around shared centers—enabling precise within-group density modeling and flexible across-group information sharing. The model permits analytical derivation of prior moments, inter-group correlations, and posterior distributions, thereby mitigating fragmentation mechanistically. Coupled with a conditional MCMC algorithm, it supports efficient marginal and predictive inference. In simulations and large-scale galaxy data analyses, our method significantly improves robustness of cross-group clustering and density estimation accuracy—both under correct specification and model misspecification.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Clustering
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Clustering observations across partially exchangeable groups of data is a routine task in Bayesian nonparametrics. Previously proposed models allow for clustering across groups by sharing atoms in the group-specific mixing measures. However, exact atom sharing can be overly rigid when groups differ subtly, introducing a trade-off between clustering and density estimates and fragmenting across-group clusters, particularly at larger sample sizes. We introduce the hierarchical shot-noise Cox process (HSNCP) mixture model, where group-specific atoms concentrate around shared centers through a kernel. This enables accurate density estimation within groups and flexible borrowing across groups, overcoming the density-clustering trade-off of previous approaches. Our construction, built on the shot-noise Cox process, remains analytically tractable: we derive closed-form prior moments and an inter-group correlation, obtain the marginal law and predictive distribution for latent parameters, as well as the posterior of the mixing measures given the latent parameters. We develop an efficient conditional MCMC algorithm for posterior inference. We assess the performance of the HSNCP model through simulations and an application to a large galaxy dataset, demonstrating balanced across-group clusters and improved density estimates compared with the hierarchical Dirichlet process, including under model misspecification.

Problem

• Overcoming rigid atom sharing limitations in cross-group clustering models
• Resolving the trade-off between clustering accuracy and density estimation
• Enabling flexible information borrowing across partially exchangeable data groups

Innovation

• Hierarchical shot-noise Cox process mixture model
• Kernel-based atom concentration for flexible borrowing
• Analytically tractable with efficient conditional MCMC inference