Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models

2026-08-06

AI Summary

Test-time adaptation (TTA) improves the accuracy of vision-language models under distribution shifts but often degrades calibration due to a phenomenon termed “prediction-preserving sharpening,” wherein predictions remain unchanged while confidence becomes excessively high and predictive entropy is overly compressed. This work is the first to establish the link between this phenomenon and calibration error, and introduces Zero-shot Anchor Entropy Calibration (ZAEC)—a label-free, parameter-free post-processing method that leverages zero-shot entropy as a reference for uncertainty and selectively applies temperature scaling to restore predictive entropy without altering class rankings or accuracy. Evaluated across five TTA methods and fifteen datasets on both ViT-B/16 and RN50 architectures, ZAEC consistently achieves substantial reductions in macro-averaged Expected Calibration Error (ECE), yielding state-of-the-art calibration performance.

Abstract

Test-time adaptation (TTA) can improve the recognition accuracy of vision-language models under distribution shift, but often degrades calibration, making predictive confidence unreliable for downstream decision-making. Many existing label-free calibration approaches are either coupled to prompt optimization or rely on logit-range statistics that provide only a coarse characterization of the predictive distribution. We show that TTA can increase confidence and reduce entropy even when the top-1 prediction and its correctness remain unchanged, a failure mode we term prediction-preserving sharpening. Across diverse TTA methods and benchmarks, larger entropy reductions relative to paired zero-shot predictions are associated with greater increases in Expected Calibration Error (ECE). On entropy-reduced samples, confidence gains also tend to exceed accuracy gains. Based on these findings, we propose Zero-Shot-Anchored Entropy Calibration (ZAEC), a label-free post-hoc method that uses zero-shot entropy as a sample-specific uncertainty reference. ZAEC selectively restores the zero-shot entropy of sharpened predictions through minimal temperature scaling while leaving all other predictions unchanged. It requires no labeled calibration data or learned parameters and preserves class rankings and classification accuracy. Across five TTA methods and 15 datasets, ZAEC achieves the lowest post-hoc macro-average ECE on ViT-B/16, with consistent gains on RN50.

Problem

• test-time adaptation
• calibration
• vision-language models
• distribution shift
• predictive uncertainty

Innovation

• test-time adaptation
• calibration
• zero-shot entropy
• vision-language models
• temperature scaling