Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling

2026-04-05

AI Summary

This work addresses the limitations of standard Stein Variational Gradient Descent (SVGD), which employs a single global step size and struggles to balance the disparate evolution rates of attraction and repulsion effects in high-dimensional, anisotropic, or hierarchical posteriors, often leading to inefficiency or instability. To overcome this, the paper introduces a multi-timescale perspective into SVGD for the first time, proposing a multi-rate SVGD framework that decouples the updates of attractive and repulsive dynamics. It develops three strategies: a symmetric splitting scheme, a fixed multi-rate variant (MR-SVGD), and an adaptive multi-rate version (Adapt-MR-SVGD). Empirical evaluations across six benchmark tasks demonstrate substantial improvements over standard SVGD, particularly in rigid hierarchical models, strongly anisotropic distributions, and multimodal settings, with the adaptive variant achieving the best performance and the fixed variant offering a computationally efficient and robust alternative.

Abstract

Many particle-based Bayesian inference methods use a single global step size for all parts of the update. In Stein variational gradient descent (SVGD), however, each update combines two qualitatively different effects: attraction toward high-posterior regions and repulsion that preserves particle diversity. These effects can evolve at different rates, especially in high-dimensional, anisotropic, or hierarchical posteriors, so one step size can be unstable in some regions and inefficient in others. We derive a multirate version of SVGD that updates these components on different time scales. The framework yields practical algorithms, including a symmetric split method, a fixed multirate method (MR-SVGD), and an adaptive multirate method (Adapt-MR-SVGD) with local error control. We evaluate the methods in a broad and rigorous benchmark suite covering six problem families: a 50D Gaussian target, multiple 2D synthetic targets, UCI Bayesian logistic regression, multimodal Gaussian mixtures, Bayesian neural networks, and large-scale hierarchical logistic regression. Evaluation includes posterior-matching metrics, predictive performance, calibration quality, mixing, and explicit computational cost accounting. Across these six benchmark families, multirate SVGD variants improve robustness and quality-cost tradeoffs relative to vanilla SVGD. The strongest gains appear on stiff hierarchical, strongly anisotropic, and multimodal targets, where adaptive multirate SVGD is usually the strongest variant and fixed multirate SVGD provides a simpler robust alternative at lower cost.

Problem

• Stein variational gradient descent
• multirate
• Bayesian sampling
• anisotropic posteriors
• hierarchical models

Innovation

• Multirate SVGD
• Bayesian inference
• Particle-based sampling
• Adaptive time stepping
• Posterior approximation