Guiding Posterior Exploration with Optimizer-Derived Geometry

2026-07-28

AI Summary

Sampling from high-dimensional, multimodal posterior distributions often hinders the efficiency of uncertainty quantification in Bayesian neural networks due to high computational costs and slow convergence. This work proposes a novel preconditioned Markov chain Monte Carlo (MCMC) sampling strategy that leverages curvature information naturally generated by adaptive optimizers—such as AdamW—during the warm-up phase to guide posterior exploration. The approach incurs no additional computational overhead and substantially shortens or even eliminates the burn-in period while enhancing numerical stability. Empirical evaluations across diverse datasets and network architectures demonstrate that the method significantly accelerates sampling convergence without compromising—and sometimes even improving—predictive performance and the quality of uncertainty estimates.

Abstract

Sampling-based methods offer a principled approach to uncertainty quantification in Bayesian neural networks. Their practical use, however, is often challenged by the computational cost of exploring high-dimensional and multimodal posterior distributions. To overcome these difficulties, Bayesian Deep Ensembles, i.e., warmstarting the sampling from several optimized solutions, have proven to be an effective strategy. In this paper, we demonstrate that curvature estimates computed during the warmstart as a byproduct in adaptive optimizers such as AdamW can inform the sampling phase at negligible additional cost. Specifically, our proposed preconditioned sampling strategy based on optimizer-derived geometries can substantially reduce or even eliminate the need for a lengthy sampling burn-in phase and leads to greater numerical stability. This approach consistently maintains or improves predictive performance and uncertainty quantification without any additional computational costs. We confirm the consistency of our findings across various datasets and network architectures.

Problem

• Bayesian neural networks
• posterior exploration
• uncertainty quantification
• sampling-based methods
• computational cost

Innovation

• optimizer-derived geometry
• preconditioned sampling
• Bayesian neural networks
• adaptive optimizers
• uncertainty quantification