Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance

2026-07-28

AI Summary

This work addresses the common reliance on Gaussian likelihoods in Bayesian neural networks, which can impair regression performance due to model misspecification. The study systematically evaluates a range of non-Gaussian likelihoods—particularly the Student’s t-distribution—within a variational inference framework. For the first time, it demonstrates across diverse data distributions and multilayer perceptron architectures that the Student’s t likelihood consistently outperforms the Gaussian assumption. Experimental results show that this approach not only yields significantly higher predictive accuracy but also reduces training time in certain scenarios. Moreover, its straightforward implementation fills a notable gap in the literature concerning likelihood selection for Bayesian neural networks.

Abstract

In Bayesian neural networks (BNNs), variational inference is a widely adopted framework for modeling uncertainty in a distributional way, with the evidence lower bound (ELBO) serving as the standard objective function. Several distributions contribute to the ELBO loss, such as the prior, approximated posterior, and likelihood distribution. Typically, these distributions are all approximated by a Gaussian distribution, since it is easy to compute, allows for reparameterized gradients, and provides a closed-form loss for training. However, several works have highlighted that this assumption may not generally hold, posing the risk of model misspecification. Alternative distributions have been proposed for the prior specifically, while the effect of distribution choice on the likelihood distribution remains unexplored. In this work, our aim is to close this gap by investigating whether alternative assumptions for the likelihood distribution can outperform the commonly used Gaussian. We compare several likelihood distribution assumptions, such as skewed or heavy-tailed, across regression tasks on both artificial and real-world datasets using standard multilayer perceptrons (MLPs). Our findings demonstrate that Student's t yields better predictive performance than a Gaussian likelihood distribution, independent of the data distribution and MLP architecture (depth and width). In some cases, Student's t can also lead to shorter training times, while still being easy to implement.

Problem

• Bayesian neural networks
• likelihood distribution
• model misspecification
• Student's t distribution
• Gaussian assumption

Innovation

• Student's t likelihood
• Bayesian neural networks
• variational inference
• likelihood distribution
• model misspecification