Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification

2026-07-26

AI Summary

This work addresses the limited functional diversity in Last-Layer Ensemble (LLE) methods—caused by shared backbone gradients—which degrades out-of-distribution (OOD) detection performance. To overcome this, the authors propose Covariance Last-Layer Ensemble (cov-LLE), which directly imposes a covariance penalty in function space to explicitly enhance ensemble member diversity, surpassing the indirect decorrelation achieved by conventional weight orthogonality. The method introduces a scale-invariant, label-free directional uncertainty score that substantially improves near-OOD detection. Without increasing forward-pass computational cost or compromising in-distribution accuracy, cov-LLE boosts predictive variance from 0.05 to 9.3 (approaching the 22.1 of deep ensembles), reduces expected calibration error (ECE) from 0.135 to 0.090, and consistently improves ROC AUC by 0.16–0.18 across diverse backbone architectures.

Abstract

A Last-Layer Ensemble (LLE), $K$ linear units on one shared frozen feature map, is an efficient single-pass approach to the disagreement-based epistemic uncertainty for out-of-distribution (OOD) detection. Its weakness is that members share the backbone gradient and can converge toward the same function, collapsing the inter-member diversity the signal depends on. Whether last-layer diversity can be restored, and what mitigates the collapse, is an open question. The weight-orthonormality defining Orthonormal Certificates (OC), the weight-orthonormal special case of the LLE, is only an indirect correction; it decorrelates the weights of the members, not their predictions. Here, we instead target the collapse directly in function space, with a Covariance Last-Layer Ensemble (cov-LLE) that places a direct covariance penalty on member activations. Cov-LLE restores the function-space diversity that weight-orthonormality cannot, and at matched $K$ recovers much of the diversity and calibration of a deep ensemble at $1\times$ backbone cost (in-distribution prediction variance $0.05\!\to\!9.3$ vs.\ $22.1$ ($\times10^{-3}$), and ECE $0.135\!\to\!0.090$ vs.\ $0.035$, for a $K\times$-cost deep ensemble), at no cost to accuracy. Viewing OC as a last-layer ensemble also organizes detectors into a two-axis taxonomy (by how their units are trained and how their outputs are scored) and exposes the OC score as a magnitude, motivating a scale-invariant, label-free direction score that repairs its near-OOD failure, adding $+0.16$ to $+0.18$ ROC AUC on every backbone.

Problem

• Last-Layer Ensemble
• function-space diversity
• epistemic uncertainty
• out-of-distribution detection
• ensemble collapse

Innovation

• Covariance Last-Layer Ensemble
• function-space diversity
• epistemic uncertainty
• out-of-distribution detection
• last-layer ensemble