The Curious Case of the Default Settings: Evaluating Default Performance of Variational Inference Software

2026-08-02

AI Summary

This study addresses the reliability of variational inference (VI) as used by practitioners, demonstrating that default configurations in popular probabilistic programming frameworks can yield substantially biased, unstable, or silently failing posterior approximations—even on simple one-dimensional conjugate models with known analytic solutions. Focusing on software defaults rather than algorithmic variants, the work systematically evaluates PyMC, NumPyro, and TensorFlow Probability from the perspective of a typical user. It reveals critical differences across frameworks in implementation choices such as initialization and variable transformation strategies, which significantly impact inference accuracy and stability. The findings caution against uncritical reliance on default settings, as they can produce inconsistent and misleading results despite the apparent simplicity of the underlying model.

Abstract

We systematically evaluate a suite of off-the-shelf variational inference (VI) software packages from the perspective of a standard practitioner. Using simple analytic benchmark models, we assess the accuracy and stability of the default VI settings in PyMC, NumPyro, and TensorFlow Probability. Unlike previous research focusing on methodological advances, our evaluation emphasizes software implementation and the default configurations that typical users encounter. Our results show that default settings can yield biased approximations of posterior summaries even for simple one-dimensional conjugate models, controls of initialization and transformations differ between software implementations, and relying on defaults may yield silent failures or poor approximations.

Problem

• variational inference
• default settings
• software evaluation
• posterior approximation
• bias

Innovation

• variational inference
• default settings
• software evaluation
• posterior approximation
• benchmarking