🤖 AI Summary
This work investigates the intrinsic relationships among classifier calibration, confidence quantification, and accuracy prediction under data distribution shift. Theoretically, we establish—for the first time—the computational equivalence of these three tasks, proving their mutual reducibility. Methodologically, we develop a unified cross-task adaptation framework grounded in this reducibility, systematically reusing and enhancing classical algorithms—including Platt scaling, expectation-maximization–based confidence quantification, and accuracy regression. Empirically, our general-purpose approach achieves performance on par with or superior to task-specific methods across multiple distribution-shift benchmarks (e.g., ImageNet-C, CIFAR-10-C). By unifying previously disjoint objectives, this work bridges longstanding task boundaries, offering both a coherent theoretical foundation and a practical, implementation-ready methodology for building distribution-shift-robust predictive models.
📝 Abstract
When the distribution of the data used to train a classifier differs from that of the test data, i.e., under dataset shift, well-established routines for calibrating the decision scores of the classifier, estimating the proportion of positives in a test sample, or estimating the accuracy of the classifier, become particularly challenging. This paper investigates the interconnections among three fundamental problems, calibration, quantification, and classifier accuracy prediction, under dataset shift conditions. Specifically, we prove their equivalence through mutual reduction, i.e., we show that access to an oracle for any one of these tasks enables the resolution of the other two. Based on these proofs, we propose new methods for each problem based on direct adaptations of well-established methods borrowed from the other disciplines. Our results show such methods are often competitive, and sometimes even surpass the performance of dedicated approaches from each discipline. The main goal of this paper is to fostering cross-fertilization among these research areas, encouraging the development of unified approaches and promoting synergies across the fields.