Minimaxity and Admissibility of Bayesian Neural Networks

2026-04-06

AI Summary

This work investigates the decision rules induced by deep fully connected ReLU Bayesian neural networks under a normal location model, with a focus on minimaxity and admissibility under quadratic loss. By introducing a hyperprior on the effective output variance, the authors construct a Bayes rule whose marginal density possesses a superharmonic square root. This study establishes, for the first time, a decision criterion for Bayesian neural networks that simultaneously satisfies both minimaxity and admissibility, and extends the result to predictive density estimation under Kullback–Leibler loss. Theoretical analysis confirms that the proposed rule is both optimal and admissible, and numerical experiments corroborate its empirical effectiveness.

Abstract

Bayesian neural networks (BNNs) offer a natural probabilistic formulation for inference in deep learning models. Despite their popularity, their optimality has received limited attention through the lens of statistical decision theory. In this paper, we study decision rules induced by deep, fully connected feedforward ReLU BNNs in the normal location model under quadratic loss. We show that, for fixed prior scales, the induced Bayes decision rule is not minimax. We then propose a hyperprior on the effective output variance of the BNN prior that yields a superharmonic square-root marginal density, establishing that the resulting decision rule is simultaneously admissible and minimax. We further extend these results from the quadratic loss setting to the predictive density estimation problem with Kullback--Leibler loss. Finally, we validate our theoretical findings numerically through simulation.

Problem

• Bayesian neural networks
• minimaxity
• admissibility
• decision theory
• predictive density estimation

Innovation

• Bayesian neural networks
• minimaxity
• admissibility
• superharmonic prior
• predictive density estimation