Spectral Collapsed Gibbs Sampler for Bayesian Sparse Regression

2026-05-06

AI Summary

In high-dimensional Bayesian sparse regression, Gibbs samplers suffer from poor efficiency due to slow mixing of the global scale parameter τ. This work proposes a novel approach that marginalizes out the regression coefficients when updating τ, enabling direct and efficient sampling from the collapsed posterior of τ by leveraging spectral decomposition and adaptive numerical integration—without requiring Metropolis–Hastings steps. The method achieves, for the first time, tuning-free direct sampling of τ under global–local shrinkage priors such as the horseshoe, substantially improving mixing efficiency and convergence speed. Its effectiveness and scalability are demonstrated on logistic regression tasks with dimensions as large as 120,000 × 1,379 and 1,980 × 17,848.

Abstract

Sparse regression based on global-local shrinkage priors are increasingly used for Bayesian modeling of modern high-dimensional data, but scaling up the Gibbs sampler for posterior inference remains a challenge. While much effort has gone into speeding up the high-dimensional coefficient update step, insufficient attention has been given to the potential poor mixing of the global scale parameter $τ$ and of the overall sampler. One proposed remedy has been to marginalize out the coefficients when updating $τ$. Here we show that, while this collapsed update was previously thought to require a Metropolis step, we can in fact sample directly and efficiently from the collapsed density. This is made possible by careful linear algebraic manipulations and a strategic per-Gibbs-scan spectral decomposition, allowing subsequent evaluations of the collapsed density across hundreds of values of $τ$ at negligible cost. We combine this computational trick with adaptive numerical integration and inverse transform sampling to construct a direct sampler. This eliminates the need to tune Metropolis proposals and yields faster convergence and improved mixing. We demonstrate our method on two big data applications, fitting logistic regression under the horseshoe prior to datasets with design matrices of size 120,000 x 1,379 and 1,980 x 17,848.

Problem

• Bayesian sparse regression
• Gibbs sampler
• global-local shrinkage priors
• poor mixing
• global scale parameter

Innovation

• Collapsed Gibbs Sampling
• Spectral Decomposition
• Global-Local Shrinkage Priors
• Bayesian Sparse Regression
• Adaptive Numerical Integration