Reliability-Aware Checkpoint Selection for Domain Generalization

2026-09-30

AI Summary

This study addresses the unreliability of target-domain predictive probabilities in domain generalization when model checkpoints are selected solely based on source validation accuracy. We propose an accuracy-constrained reliability selection method that, within a fixed training trajectory, first filters a near-optimal accuracy candidate set and then aggregates normalized negative log-likelihood with class-wise calibration error to rank candidates by the infinity norm for optimal checkpoint reselection. This approach requires neither additional training nor weight averaging. Empirical results demonstrate that it significantly reduces the expected calibration error (ECE) and negative log-likelihood (NLL) on target domains while marginally improving average target-domain accuracy, thereby effectively enhancing the overall predictive reliability of the model.

Abstract

Checkpoint selection in domain generalization often relies on source-validation accuracy, yet the selected checkpoint need not provide reliable probabilities on unseen target domains. Source-target distribution shifts can alter accuracy rankings, while accuracy alone does not measure predictive probability quality. We identify an empirical selection opportunity within fixed training trajectories: reselecting among checkpoints with near-optimal source accuracy can improve mean target probability quality with small observed changes in mean target accuracy. We study accuracy-constrained reliability selection (AC), which retains checkpoints within a tolerance of the best source-validation accuracy and ranks them by source reliability. Our reference rule aggregates within-set normalized negative log-likelihood (NLL) and class-wise calibration error (CwECE) using $D_\infty$. AC uses no target data and requires neither additional training nor weight averaging. We evaluate five domain generalization training algorithms on three benchmarks, using PACS to develop the objectives and a 0.5-percentage-point tolerance. In exploratory aggregation comparisons on 360 OfficeHome and TerraIncognita runs, the reference rule reduces mean target soft-bin squared-gap ECE and CwECE by 0.240% and 0.182%, respectively, and NLL by 0.030 relative to Source-Acc. Mean target accuracy changes by +0.213 percentage points. These results identify opportunities for reliability-aware reselection, while the additional benefit of joint over single-objective ranking remains unresolved.

Problem

• Domain Generalization
• Checkpoint Selection
• Predictive Reliability
• Probability Calibration
• Distribution Shift

Innovation

• Domain Generalization
• Checkpoint Selection
• Reliability-Aware
• Calibration Error
• Accuracy-Constrained