Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes

2025-05-21

AI Summary

This paper addresses the challenge of dynamically optimizing Bayesian model ensembles under online continual learning. We propose Online Bayesian Stacking (OBS), a method that adaptively aggregates multiple Bayesian predictors by online maximization of the logarithmic score. Our key contributions are threefold: (i) we establish, for the first time, a theoretical connection between OBS and portfolio selection, revealing a fundamental distinction in optimization objectives between OBS and online Bayesian Model Averaging (BMA); (ii) we derive a falsifiable, practical criterion for ensemble weight selection; and (iii) we provide a rigorous regret bound guaranteeing statistical consistency. Empirical results demonstrate that OBS significantly outperforms online BMA under non-stationary data streams and model misspecification, while yielding interpretable performance bounds. Collectively, OBS establishes a novel, robust paradigm for deploying Bayesian ensembles in streaming-data settings.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We revisit the classical problem of Bayesian ensembles and address the challenge of learning optimal combinations of Bayesian models in an online, continual learning setting. To this end, we reinterpret existing approaches such as Bayesian model averaging (BMA) and Bayesian stacking through a novel empirical Bayes lens, shedding new light on the limitations and pathologies of BMA. Further motivated by insights from online optimization, we propose Online Bayesian Stacking (OBS), a method that optimizes the log-score over predictive distributions to adaptively combine Bayesian models. A key contribution of our work is establishing a novel connection between OBS and portfolio selection, bridging Bayesian ensemble learning with a rich, well-studied theoretical framework that offers efficient algorithms and extensive regret analysis. We further clarify the relationship between OBS and online BMA, showing that they optimize related but distinct cost functions. Through theoretical analysis and empirical evaluation, we identify scenarios where OBS outperforms online BMA and provide principled guidance on when practitioners should prefer one approach over the other.

Problem

• Learning optimal combinations of Bayesian models online
• Reinterpreting Bayesian model averaging via empirical Bayes
• Connecting Bayesian ensemble learning with portfolio selection

Innovation

• Online Bayesian Stacking optimizes log-score adaptively
• Connects Bayesian ensembles with portfolio selection theory
• Clarifies OBS and online BMA cost function differences