Variational Bayesian Bow tie Neural Networks with Shrinkage

2024-11-17

arXiv.org

AI Summary

Deep learning models often suffer from overconfident predictions, poor adversarial robustness, and underestimated uncertainty. To address these issues, we propose a variational Bayesian Bow-tie neural network equipped with shrinkage priors (e.g., the horseshoe prior), which relaxes conventional variational inference assumptions—namely, layer-wise independence and restrictive parametric posterior families. Our method leverages Polya-Gamma data augmentation to enable conditionally linear-Gaussian approximations and integrates sparse priors to drive data-adaptive architecture learning. Empirically, it achieves superior uncertainty calibration and enhanced adversarial robustness, outperforming state-of-the-art Bayesian neural networks on standard benchmarks. Moreover, it attains significantly higher inference efficiency than MCMC-based approaches and exhibits strong robustness to architectural choices. The core innovation lies in decoupling structural design from variational assumptions, thereby enabling high-accuracy, high-efficiency, and high-robustness Bayesian deep learning.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Despite the dominant role of deep models in machine learning, limitations persist, including overconfident predictions, susceptibility to adversarial attacks, and underestimation of variability in predictions. The Bayesian paradigm provides a natural framework to overcome such issues and has become the gold standard for uncertainty estimation with deep models, also providing improved accuracy and a framework for tuning critical hyperparameters. However, exact Bayesian inference is challenging, typically involving variational algorithms that impose strong independence and distributional assumptions. Moreover, existing methods are sensitive to the architectural choice of the network. We address these issues by constructing a relaxed version of the standard feed-forward rectified neural network, and employing Polya-Gamma data augmentation tricks to render a conditionally linear and Gaussian model. Additionally, we use sparsity-promoting priors on the weights of the neural network for data-driven architectural design. To approximate the posterior, we derive a variational inference algorithm that avoids distributional assumptions and independence across layers and is a faster alternative to the usual Markov Chain Monte Carlo schemes.

Problem

• Address overconfident predictions in deep models
• Enhance robustness to adversarial attacks
• Reduce sensitivity to architectural network choices

Innovation

• Stochastic relaxation of feed-forward neural networks
• Sparsity-promoting priors for robust architecture
• Polya-Gamma data augmentation for fast inference