Can Bayesian Neural Networks Make Confident Predictions?

2025-01-20

AI Summary

Bayesian neural networks (BNNs) face fundamental challenges in modeling predictive uncertainty: the multimodality of the posterior predictive distribution is difficult to capture, unimodal approximations often fail, and there lacks a quantitative characterization of how learning capacity relates to architectural and data-scale parameters. Method: We rigorously derive, for the first time under discretized inner-layer weight priors, a Gaussian mixture-form posterior predictive distribution; uncover intrinsic links between parameter equivalence classes and network scaling mechanisms; and systematically identify, quantify, and interpret predictive multimodality. Leveraging Bayesian inference, Gaussian mixture modeling, and scaling-law analysis—incorporating sample size, layer width, and output dimension—we establish precise failure conditions for unimodal approximations. Contribution: We derive an explicit quantitative relationship between posterior predictive contraction (i.e., learning capacity) and network scaling, providing both theoretical foundations and practical criteria for BNN uncertainty calibration.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Bayesian inference promises a framework for principled uncertainty quantification of neural network predictions. Barriers to adoption include the difficulty of fully characterizing posterior distributions on network parameters and the interpretability of posterior predictive distributions. We demonstrate that under a discretized prior for the inner layer weights, we can exactly characterize the posterior predictive distribution as a Gaussian mixture. This setting allows us to define equivalence classes of network parameter values which produce the same likelihood (training error) and to relate the elements of these classes to the network's scaling regime -- defined via ratios of the training sample size, the size of each layer, and the number of final layer parameters. Of particular interest are distinct parameter realizations that map to low training error and yet correspond to distinct modes in the posterior predictive distribution. We identify settings that exhibit such predictive multimodality, and thus provide insight into the accuracy of unimodal posterior approximations. We also characterize the capacity of a model to"learn from data"by evaluating contraction of the posterior predictive in different scaling regimes.

Problem

• Bayesian Neural Networks
• Uncertainty Quantification
• Posterior Distribution

Innovation

• Discrete Prior Weights
• Gaussian Mixture Model Posterior
• Network Scalability Analysis