Posterior Mode Guided Dimension Reduction for Bayesian Model Averaging in Heavy-Tailed Linear Regression

2026-02-24

AI Summary

This work addresses the challenges in high-dimensional Bayesian regression, where conventional MCMC methods often get trapped in local modes and maximum a posteriori (MAP) estimation fails to quantify uncertainty. To overcome these limitations, the authors propose a hybrid approach that integrates deterministic optimization with stochastic sampling. Specifically, under a heavy-tailed hyperbolic error model, they first employ a two-stage ECM algorithm to efficiently perform variable selection and substantially reduce the model space. Subsequently, Gibbs sampling is conducted within the high posterior probability subspace to enable full posterior inference, complemented by Bayesian model averaging. The proposed method effectively balances computational efficiency, variable selection accuracy, and robust uncertainty quantification. Empirical evaluations on both simulated and real-world datasets demonstrate its superior performance over current state-of-the-art methods.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

For large model spaces, the potential entrapment of Markov chain Monte Carlo (MCMC) based methods with spike-and-slab priors poses significant challenges in posterior computation in regression models. On the other hand, maximum a posteriori (MAP) estimation, which is a more computationally viable alternative, fails to provide uncertainty quantification. To address these problems simultaneously and efficiently, this paper proposes a hybrid method that blends MAP estimation with MCMC-based stochastic search algorithms within a heavy-tailed error framework. Under hyperbolic errors, the current work develops a two-step expectation conditional maximization (ECM) guided MCMC algorithm. In the first step, we conduct an ECM-based posterior maximization and perform variable selection, thereby identifying a reduced model space in a high posterior probability region. In the second step, we execute a Gibbs sampler on the reduced model space for posterior computation. Such a method is expected to improve the efficiency of posterior computation and enhance its inferential richness. Through simulation studies and benchmark real life examples, our proposed method is shown to exhibit several advantages in variable selection and uncertainty quantification over various state-of-the-art methods.

Problem

• Bayesian model averaging
• heavy-tailed regression
• posterior computation
• variable selection
• uncertainty quantification

Innovation

• Bayesian model averaging
• heavy-tailed regression
• ECM-guided MCMC
• posterior mode
• dimension reduction