Reversible Jump MCMC With No Regrets: Bayesian Variable Selection Using Mixtures of Mutually Singular Distributions

2026-04-30

AI Summary

This study addresses the challenge of jointly sampling discrete model indicators and continuous parameters in Bayesian variable selection. The authors propose a Mixture of Mutually Singular distributions (MoMS) framework that reformulates variable selection as a sampling problem over mutually singular subspaces within a fixed-dimensional parameter space. This approach preserves the interpretability of spike-and-slab priors while avoiding trans-dimensional jumps. MoMS requires only standard fixed-dimensional MCMC and Metropolis–Hastings algorithms, yielding an implementation that is both straightforward and provably equivalent to reversible jump MCMC (RJMCMC) in terms of acceptance probabilities. Empirical results demonstrate that MoMS exactly reproduces full-enumeration posterior inclusion probabilities on benchmark datasets, achieves higher effective sample sizes than carefully tuned RJMCMC, and successfully extends to mixed-effects logistic regression and factor loading selection tasks.

Abstract

Bayesian variable selection requires sampling from a posterior distribution that combines discrete model indicators with continuously varying parameters, a challenge often addressed through reversible jump Markov chain Monte Carlo (RJMCMC). Despite its generality, RJMCMC is widely regarded as difficult to design and implement correctly. We present mixtures of mutually singular (MoMS) distributions as a transparent alternative in which competing models are represented within a single fixed-dimensional parameter space partitioned into mutually singular subspaces. We show that this formulation reproduces the exact spike-and-slab interpretation of Bayesian variable selection and that, under appropriate constructions, MoMS and RJMCMC share the same Metropolis--Hastings acceptance probability. On a benchmark dataset with ten predictors, both methods recover posterior inclusion probabilities that match full enumeration, while MoMS achieves comparable or superior effective sample size per second relative to a carefully engineered RJMCMC scheme. We further illustrate the approach in a mixed-effects logistic regression for a sleep-and-memory experiment and in factor-loading selection for a multidimensional generalized partial credit model. Together, these results show that Bayesian variable selection can be carried out within standard fixed-dimensional Markov chain Monte Carlo methodology -- without regret.

Problem

• Bayesian variable selection
• Reversible Jump MCMC
• posterior sampling
• discrete-continuous parameters
• model uncertainty

Innovation

• Mixtures of Mutually Singular
• Bayesian variable selection
• Reversible Jump MCMC
• spike-and-slab
• fixed-dimensional MCMC