🤖 AI Summary
Soft-label Bayesian error estimators are prone to severe distortion under non-genuine posteriors, potentially misestimating the irreducible error even when labels are calibrated. This work systematically analyzes the influence of temperature scaling on proxy measures of the Bayes error, providing for the first time an exact analytical expression for this distortion. We prove that the proxy value varies strictly monotonically with temperature and can be arbitrarily adjusted without altering classification performance. Under a Gaussian logits assumption, we derive a closed-form two-parameter solution. Experiments across eight binary classification tasks on CIFAR-10, Fashion-MNIST, and SVHN show that, while test error remains constant, the proxy value can vary by factors of 56 to 980; the closed-form estimates deviate from empirical values by less than 0.018, and the temperature minimizing expected calibration error does not correspond to a stable proxy estimate.
📝 Abstract
The soft-label Bayes-error estimator beta(z) = E[min(z, 1-z)] of Ishida et al. estimates the irreducible error of a binary task directly from probability-valued labels. Recent work by Ushio et al. showed that this estimator is fragile when the probabilities are not the true posterior: even perfectly calibrated soft labels can yield a substantially inaccurate estimate, and they propose isotonic calibration as a consistent remedy. We complement that line of work by characterizing exactly how the most widely used post-hoc calibration map -- temperature scaling -- distorts the proxy. We prove an exact, model-free identity reducing the temperature-scaled proxy to the classifier's margin distribution, from which we obtain (i) strict monotonicity in the temperature and (ii) a continuous bijection from the temperature axis onto the open interval (0, 1/2), so that a fixed classifier -- with fixed decisions and fixed 0-1 error -- can be made to report any proxy value whatsoever. Under a Gaussian model of the logits we further derive a two-parameter closed form for the entire proxy-versus-temperature curve. Across CIFAR-10, Fashion-MNIST, and SVHN (eight binary tasks), the proxy varies by 56x to 980x at constant test error, the closed form reproduces the empirical curve to within 0.018, and the calibration temperature that minimizes the expected calibration error does not coincide with any stable proxy value. Our results give a precise, predictive account of the distortion whose existence motivates calibration-based remedies, and they reinforce the practical recommendation that a proxy value is meaningful only together with the mechanism that produced its probabilities.