🤖 AI Summary
This work addresses the limited predictive diversity and poor out-of-distribution (OOD) uncertainty quantification of single pretrained models under distribution shift. The authors propose a Perturb-and-Correct approach that constructs a posterior ensemble using only a single model by applying random perturbations to hidden layers of the pretrained network and subsequently correcting them via least-squares affine transformations. This method uniquely exploits the affine redundancy inherent in neural networks, enhancing OOD prediction diversity and uncertainty calibration without compromising in-distribution performance. Empirical evaluations demonstrate that the proposed technique achieves a superior or competitive trade-off between in-distribution accuracy and OOD detection on MuJoCo dynamics prediction and CIFAR-10 OOD benchmarks compared to existing posterior ensemble baselines.
📝 Abstract
Models that are indistinguishable on in-distribution data can behave very differently under distribution shift. We introduce Perturb-and-Correct (P&C), a post-hoc method for constructing epistemically diverse predictors from a single pretrained network. P&C applies random hidden layer perturbations with a least-squares correction in the subsequent affine layer, producing predictors that agree on calibration data while remaining free to disagree away from it. We analyze this mechanism through the post-correction residual and its first-order sensitivity: the residual is controlled near the calibration distribution by a leverage term, while corrected sensitivity grows as inputs deviate from the calibration geometry. Empirically, P&C achieves a strong ID/OOD tradeoff across MuJoCo dynamics prediction and CIFAR-10 OOD detection, matching or outperforming standard post-hoc baselines while requiring only a single pretrained model. Our findings highlight the potential in further exploiting overparameterization as a strength of deep learning models.