🤖 AI Summary
This study addresses the limitation of existing posterior calibration evaluation practices, which predominantly focus on aggregate performance while overlooking robustness across varying operational conditions within datasets. The authors propose the first conditionally stratified evaluation framework, employing preregistered experiments to systematically compare the robustness of temperature scaling (TEMP) and isotonic regression (ISO) under four controlled conditions. The assessment encompasses multiple dimensions—including calibration slope, Brier score, discrimination, and AUROC—and applies Holm’s correction for multiplicity in rigorous hypothesis testing. Results demonstrate that TEMP consistently achieves calibration slopes closer to one and superior, more stable Brier scores across all conditions; differences in discrimination between the two methods are negligible; and AUROC performance varies by condition, revealing that the robustness of calibration methods is highly dependent on both specific operational contexts and the choice of evaluation metric.
📝 Abstract
Post-hoc calibration is widely adopted to correct probability estimates from trained classifiers, yet most evaluations report aggregate performance without testing whether that performance holds across distinct operating conditions within a single dataset. We present a pre-registered, condition-stratified robustness analysis comparing temperature scaling (TEMP) and isotonic regression (ISO) across four controlled conditions (C1--C4). Four hypothesis groups are evaluated: discrimination deltas with Holm-corrected multiplicity control (H1), Brier score differences (H2), calibration slope outcomes (H3), and AUROC differences under best-condition setups (H4). TEMP-minus-ISO discrimination deltas remain small across all conditions (-0.0155 to 0.0139), with Holm-adjusted p-values of 0.9895 everywhere. TEMP Brier differences are consistently negative (C1: -0.0002 through C4: -0.0074), while ISO shows sign reversals. TEMP calibration slopes stay closer to unity in every condition (range 0.7597--0.9493) than ISO slopes (0.1364--0.2726). AUROC differences shift from near zero in C1 (-0.0004) to positive in C4 (0.0264). These results establish that in-dataset robustness is condition-dependent and metric-specific. No claim of external transportability is made.