Is the Last Layer Sufficient for Uncertainty Quantification?

2026-05-28

AI Summary

This study addresses the challenge of efficiently and reliably quantifying epistemic uncertainty in deep neural networks for safety-critical applications, where existing Bayesian approaches often incur prohibitive computational costs. By leveraging random matrix theory and extensive empirical evaluation, the work systematically compares Bayesian generalized linear models constructed via full-network linearization against those using only the final-layer linearization. The analysis demonstrates, for the first time, that linearizing solely the last layer yields uncertainty estimates nearly on par with full-network linearization while achieving substantially improved computational efficiency. This finding provides both theoretical justification and practical guidance for deploying scalable Bayesian deep learning methods in real-world settings.

Abstract

Epistemic uncertainty quantification (UQ) for deep neural networks (DNNs) is a requirement for safe adoption of AI in mission-critical settings. Several leading methods for UQ linearize DNNs to form Bayesian Generalized Linear Models (GLMs), where epistemic uncertainty is modeled via the predictive posterior distribution. Linearizing around the parameters of the final connected layer of a DNN is a commonly used approximation for reducing the computational burden of such GLMs, though it is often believed to come at the cost of degraded performance. In this work, we compare GLMs arising from full-network and last-layer linearization using both theoretical and empirical approaches. We first employ tools from random matrix theory to conduct a theoretical comparison; this analysis reveals no meaningful improvement in the UQ capabilities of full linearization. Coupled with a large-scale empirical evaluation across a range of modern machine learning tasks, we arrive at the following conclusion: a last-layer approximation yields comparable UQ performance while offering substantially improved computational efficiency.

Problem

• epistemic uncertainty quantification
• deep neural networks
• last-layer linearization
• Bayesian Generalized Linear Models
• computational efficiency

Innovation

• epistemic uncertainty quantification
• last-layer linearization
• Bayesian GLMs
• random matrix theory
• computational efficiency