🤖 AI Summary
Deep learning models often suffer from miscalibrated predictive uncertainty: their predicted confidence intervals exhibit coverage rates substantially below nominal levels (e.g., 90% intervals covering <90% of test outcomes) and degraded sharpness. This work proposes a probabilistic recalibration training paradigm based on low-dimensional density estimation, jointly improving calibration and sharpness without compromising overall model performance. It is the first general-purpose distributional calibration method applicable to arbitrary models—including deep neural networks—with theoretical guarantees and a consistent convergence bound. The approach integrates maximum-likelihood enhancement with Bayesian modeling principles, yielding significant improvements in both calibration and predictive accuracy for linear and deep Bayesian models; notably, 90% confidence intervals achieve coverage rates approaching the nominal level. The implementation, including code and tutorials, is publicly available.
📝 Abstract
Accurate probabilistic predictions can be characterized by two properties -- calibration and sharpness. However, standard maximum likelihood training yields models that are poorly calibrated and thus inaccurate -- a 90% confidence interval typically does not contain the true outcome 90% of the time. This paper argues that calibration is important in practice and is easy to maintain by performing low-dimensional density estimation. We introduce a simple training procedure based on recalibration that yields calibrated models without sacrificing overall performance; unlike previous approaches, ours ensures the most general property of distribution calibration and applies to any model, including neural networks. We formally prove the correctness of our procedure assuming that we can estimate densities in low dimensions and we establish uniform convergence bounds. Our results yield empirical performance improvements on linear and deep Bayesian models and suggest that calibration should be increasingly leveraged across machine learning. We release a library that implements our methods along with a blog post here: https://shachideshpande.github.io/blog-distribution-calibration/.
