Stability-based Generalization Bounds for Variational Inference

2025-02-17

AI Summary

Existing generalization bounds fail to characterize approximate Bayesian inference algorithms—such as variational inference (VI)—that rely on stochastic gradient descent (SGD), leaving a critical theoretical gap. Method: This work systematically introduces algorithmic stability analysis into the approximate Bayesian inference framework. By quantifying how parameter perturbations during SGD affect the posterior approximation, it derives tight, verifiable, algorithm-specific generalization error bounds. Contribution/Results: The proposed bounds bridge the gap between traditional PAC-Bayes bounds and purely stability- or mutual-information-based bounds, while avoiding strong distributional or structural assumptions. Experiments on Bayesian deep learning models and real-world datasets demonstrate that the bounds are non-vacuous and effectively discriminate the generalization performance of different VI algorithms. This establishes a novel theoretical paradigm for analyzing approximate Bayesian inference methods.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Other Foundations of Computer Vision

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Variational inference (VI) is widely used for approximate inference in Bayesian machine learning. In addition to this practical success, generalization bounds for variational inference and related algorithms have been developed, mostly through the connection to PAC-Bayes analysis. A second line of work has provided algorithm-specific generalization bounds through stability arguments or using mutual information bounds, and has shown that the bounds are tight in practice, but unfortunately these bounds do not directly apply to approximate Bayesian algorithms. This paper fills this gap by developing algorithm-specific stability based generalization bounds for a class of approximate Bayesian algorithms that includes VI, specifically when using stochastic gradient descent to optimize their objective. As in the non-Bayesian case, the generalization error is bounded by by expected parameter differences on a perturbed dataset. The new approach complements PAC-Bayes analysis and can provide tighter bounds in some cases. An experimental illustration shows that the new approach yields non-vacuous bounds on modern neural network architectures and datasets and that it can shed light on performance differences between variant approximate Bayesian algorithms.

Problem

• Develops stability-based generalization bounds for VI.
• Addresses gaps in existing bounds for Bayesian algorithms.
• Applies to stochastic gradient descent optimized objectives.

Innovation

• Stability-based generalization bounds
• Stochastic gradient descent optimization
• Approximate Bayesian algorithms analysis