Bayesian Neural Network via Stochastic Gradient Descent

2020-06-04

arXiv.org

AI Summary

This work addresses the challenges of poor convergence and limited scalability in variational inference for Bayesian neural networks (BNNs). We propose a practical training framework that optimizes the evidence lower bound (ELBO) using stochastic gradient descent (SGD). For the first time, we systematically establish the theoretical feasibility of SGD for variational learning in BNNs and design a robust gradient estimation strategy to enable parameterized posterior approximation. By avoiding traditional complex sampling schemes or second-order optimization methods, our approach significantly reduces computational overhead. Experiments on five UCI regression benchmarks demonstrate that our method outperforms state-of-the-art BNN approaches in both root mean squared error (RMSE) and negative log-likelihood (NLL), thereby improving the accuracy and practicality of uncertainty quantification.

Abstract

The goal of bayesian approach used in variational inference is to minimize the KL divergence between variational distribution and unknown posterior distribution. This is done by maximizing the Evidence Lower Bound (ELBO). A neural network is used to parametrize these distributions using Stochastic Gradient Descent. This work extends the work done by others by deriving the variational inference models. We show how SGD can be applied on bayesian neural networks by gradient estimation techniques. For validation, we have tested our model on 5 UCI datasets and the metrics chosen for evaluation are Root Mean Square Error (RMSE) error and negative log likelihood. Our work considerably beats the previous state of the art approaches for regression using bayesian neural networks.

Problem

• Improves salient object detection through multi-scale context and attention mechanisms
• Enhances boundary delineation with edge-aware decoding and uncertainty modeling
• Advances prediction accuracy using boundary-sensitive and topology-preserving loss functions

Innovation

• Multi-scale context aggregation with attention mechanisms
• Edge-aware decoder with uncertainty modeling
• Boundary-sensitive and topology-preserving loss functions