bayesian deep learning

Bayesian neural networks (BNNs) face fundamental challenges in modeling predictive uncertainty: the multimodality of the posterior predictive distribution is difficult to capture, unimodal approximations often fail, and there lacks a quantitative characterization of how learning capacity relates to architectural and data-scale parameters. Method: We rigorously derive, for the first time under discretized inner-layer weight priors, a Gaussian mixture-form posterior predictive distribution; uncover intrinsic links between parameter equivalence classes and network scaling mechanisms; and systematically identify, quantify, and interpret predictive multimodality. Leveraging Bayesian inference, Gaussian mixture modeling, and scaling-law analysis—incorporating sample size, layer width, and output dimension—we establish precise failure conditions for unimodal approximations. Contribution: We derive an explicit quantitative relationship between posterior predictive contraction (i.e., learning capacity) and network scaling, providing both theoretical foundations and practical criteria for BNN uncertainty calibration.

This study systematically evaluates the quality of uncertainty estimation in binary classification models, focusing on two core objectives: probabilistic calibration and out-of-distribution (OOD) detection. Within an approximate Bayesian inference framework, we empirically compare six prominent methods—neural network ensembles, conflicting loss, evidential deep learning, Monte Carlo Dropout, Gaussian process classification, and Dirichlet process mixture models—on controlled synthetic datasets. Results show that all methods achieve strong calibration performance; however, only generative approaches (e.g., Gaussian processes, Dirichlet processes) and explicitly uncertainty-aware models (e.g., evidential deep learning) consistently enhance OOD sensitivity. In contrast, most discriminative deep learning methods exhibit inconsistent OOD responsiveness. To our knowledge, this is the first work to empirically decouple calibration from OOD robustness under a unified benchmark, providing critical empirical evidence and methodological guidance for uncertainty quantification in trustworthy machine learning.

Bayesian neural networks (BNNs) often suffer from inaccurate predictions and poorly calibrated uncertainty due to low-quality posterior approximations—especially in wide architectures and out-of-distribution (OOD) settings, where computation becomes prohibitively expensive. To address this, we systematically investigate how architectural design influences the expressiveness and exploration capability of variational posterior approximations. We propose a variational inference-based stacked-ensemble framework that significantly improves posterior coverage and predictive robustness. Experiments demonstrate that our method surpasses standard MCMC in uncertainty quantification, matches MCMC in prediction accuracy, and drastically reduces computational cost. It further achieves superior trade-offs among accuracy, uncertainty calibration, and efficiency on both wide networks and OOD data. Our core contribution lies in revealing the critical regulatory role of architecture choice in approximate Bayesian inference performance, and in providing an efficient, scalable BNN modeling paradigm.

Deep learning models often suffer from miscalibrated predictive uncertainty: their predicted confidence intervals exhibit coverage rates substantially below nominal levels (e.g., 90% intervals covering <90% of test outcomes) and degraded sharpness. This work proposes a probabilistic recalibration training paradigm based on low-dimensional density estimation, jointly improving calibration and sharpness without compromising overall model performance. It is the first general-purpose distributional calibration method applicable to arbitrary models—including deep neural networks—with theoretical guarantees and a consistent convergence bound. The approach integrates maximum-likelihood enhancement with Bayesian modeling principles, yielding significant improvements in both calibration and predictive accuracy for linear and deep Bayesian models; notably, 90% confidence intervals achieve coverage rates approaching the nominal level. The implementation, including code and tutorials, is publicly available.

This work addresses the challenge of unreliable predictive uncertainty estimation in deep neural networks, which undermines their trustworthiness in safety-critical applications. The paper presents a systematic survey of uncertainty quantification methods, with a focus on ensemble and approximate Bayesian techniques, and introduces a decoupled “method–metric” framework that unifies the generation of predictive distributions and the aggregation of uncertainties. By integrating diverse approaches—including Bayesian neural networks, Monte Carlo Dropout, deep and efficient ensembles, single-forward methods, evidential networks, conformal prediction, and post-hoc calibration—the study establishes a unified taxonomy and evaluation benchmark. This enables a clear delineation of each method’s theoretical foundations, implementation strategies, empirical performance, and limitations, while also outlining promising directions for uncertainty research in large language models.

Despite the strong predictive performance of deep neural networks, their generalization mechanisms and uncertainty quantification lack rigorous theoretical foundations. This work proposes a unified probabilistic framework that integrates Bayesian inference, function-space modeling, and large deviation theory, offering the first synthesis of diversity, smoothness, and stochasticity—the three key drivers of generalization—within the PAC-Bayes and large deviation paradigms. The core contributions include the Deep Variational Implicit Process (DVIP) model and two efficient post-hoc calibration methods, VaLLA and FMGP, which directly calibrate uncertainty estimates of pre-trained deterministic networks. These advances not only enable practical uncertainty quantification but also provide a theoretical explanation for the surprisingly good generalization observed in over-parameterized neural networks.

This work addresses the unreliability of deep learning models in high-stakes scenarios due to overconfidence, a limitation inadequately mitigated by existing Bayesian approaches that require multiple forward passes at test time and incur substantial computational overhead. To overcome this, the authors propose Calibrated Variance Propagation (CVP), an efficient method that estimates predictive uncertainty in a single forward pass and is readily applicable to modern architectures such as Transformers and CNNs. CVP introduces a novel variance propagation mechanism through normalization layers, incorporates approximations for activation functions, and applies a lightweight posterior calibration step to absorb residual errors. Experiments demonstrate that CVP substantially improves prediction coverage: on NLVR2 and VQAv2, BEiT-3 achieves coverage of 14.6% at a 0.5% risk level, up from 8.2%, while ViLT improves from 2.6% to 10.8%.

Traditional neural networks struggle to provide reliable uncertainty estimates, while Bayesian neural networks, despite their theoretical advantages, are computationally expensive and difficult to scale. This work proposes a novel approach that integrates Dirichlet distributions with Monte Carlo Dropout to structurally model predictive class probabilities, thereby yielding better-calibrated and more informative uncertainty representations while retaining the computational efficiency of Monte Carlo Dropout during inference. The method demonstrates significant improvements over existing techniques across multiple benchmark tasks, offering a practical pathway toward deploying efficient and reliable uncertainty-aware deep models in real-world applications.

This work proposes an efficient Bayesian deep ensemble method to address the limitations of existing deep ensembles in uncertainty calibration and interpretability. By leveraging low-dimensional predictive representations, independent training strategies, and a closed-form Bayesian linear regression aggregation mechanism, the approach enables analytical posterior weight inference while maintaining high predictive accuracy. This significantly enhances model interpretability and yields well-calibrated uncertainty estimates. Notably, the computational complexity shifts from scaling with dataset size to depending only on ensemble size, substantially improving scalability. Empirical evaluations on standard regression benchmarks demonstrate that the method achieves state-of-the-art predictive performance while providing reliable and properly calibrated uncertainty quantification.

This work addresses a key limitation in existing Bayesian deep learning approaches, which rely solely on a scalar mutual information metric to quantify epistemic uncertainty and thus cannot distinguish the model’s ignorance across safety-critical versus non-critical classes. The authors propose decomposing mutual information into a per-class contribution vector \( C_k(x) = \sigma_k^{2}/(2\mu_k) \), derived from the mean and variance of class probabilities over posterior samples, with a second-order Taylor expansion ensuring that the sum of these contributions approximates mutual information. This formulation enables, for the first time, interpretable and comparable per-class epistemic uncertainty estimates, mitigates boundary suppression, facilitates fair comparison between rare and common classes, and introduces a skewness-based metric to assess approximation quality. Experiments demonstrate a 34.7% reduction in risk for critical classes in diabetic retinopathy selective prediction, state-of-the-art AUROC in out-of-distribution detection—revealing asymmetric distribution shifts invisible to mutual information—and enhanced robustness to label noise.