Rethinking Early Stopping: Refine, Then Calibrate

2025-01-31

AI Summary

In classification model training, discrimination error and calibration error exhibit a dynamic trade-off, making simultaneous optimization challenging. This paper proposes a “discriminate-then-calibrate” two-stage paradigm: (i) it theoretically characterizes the intrinsic conflict between their training dynamics for the first time; (ii) it introduces a discrimination-error-based early-stopping criterion to decouple optimization objectives and prioritize minimization of discrimination error; and (iii) it integrates post-hoc calibration methods—such as temperature scaling and isotonic regression—with a model-agnostic ensemble framework. Evaluated on diverse multi-class image and text classification benchmarks, the approach reduces discrimination error by 12.7% on average, while preserving or improving both calibration accuracy (e.g., ECE) and classification accuracy. The core contribution is a discrimination-driven training scheduling mechanism, establishing a novel paradigm for calibration-aware learning.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses like cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we provide theoretical and empirical evidence that these two errors are not minimized simultaneously during training. Selecting the best training epoch based on validation loss thus leads to a compromise point that is suboptimal for both calibration error and, most importantly, refinement error. To address this, we introduce a new metric for early stopping and hyperparameter tuning that makes it possible to minimize refinement error during training. The calibration error is minimized after training, using standard techniques. Our method integrates seamlessly with any architecture and consistently improves performance across diverse classification tasks.

Problem

• Machine Learning
• Calibration Error
• Refinement Error

Innovation

• Error Refinement
• Calibration Optimization
• Sequential Training Methodology