early stopping

Existing early-exit model training paradigms—joint and separate—lack theoretical grounding and systematic empirical evaluation. Method: We formalize training mechanisms into three standardized categories: joint, separate, and hybrid; and propose a staged training strategy: first independently training the backbone, then jointly optimizing the backbone and exit classifiers. We conduct rigorous analysis via information bottleneck theory, loss curvature modeling, numerical rank estimation of activation matrices, and extensive experiments across multiple architectures (ResNet, ViT) and datasets (CIFAR, ImageNet). Contribution/Results: Our study uncovers principled patterns governing how training paradigms interact with model architecture and data characteristics. The hybrid paradigm achieves superior accuracy–latency trade-offs: up to 18% inference speedup on ImageNet with <0.3% top-1 accuracy degradation. This work establishes an interpretable, reproducible foundation for training multi-exit models, bridging theoretical insight with practical performance gains.

In sequential Bayesian experimental design, pre-specifying a fixed number of experiments fails to accommodate dynamic real-world requirements, necessitating a principled solution to the fundamental “optimal stopping” problem. This paper pioneers the integration of optimal stopping theory into this framework by formulating a Markov decision process that jointly optimizes stopping policies and experimental designs; we prove that the optimal stopping rule balances immediate reward against the expected value of continuing. To address policy circular dependency during training, we propose a curriculum learning strategy to enhance convergence stability. Our method unifies Bayesian inference, policy gradient optimization, and curriculum learning. Empirical evaluation on linear-Gaussian benchmark tasks and contaminant source localization demonstrates significant improvements in estimation accuracy and sampling efficiency over baseline methods (e.g., fixed-threshold rules), especially under strong sequential dependencies.

In classification model training, discrimination error and calibration error exhibit a dynamic trade-off, making simultaneous optimization challenging. This paper proposes a “discriminate-then-calibrate” two-stage paradigm: (i) it theoretically characterizes the intrinsic conflict between their training dynamics for the first time; (ii) it introduces a discrimination-error-based early-stopping criterion to decouple optimization objectives and prioritize minimization of discrimination error; and (iii) it integrates post-hoc calibration methods—such as temperature scaling and isotonic regression—with a model-agnostic ensemble framework. Evaluated on diverse multi-class image and text classification benchmarks, the approach reduces discrimination error by 12.7% on average, while preserving or improving both calibration accuracy (e.g., ECE) and classification accuracy. The core contribution is a discrimination-driven training scheduling mechanism, establishing a novel paradigm for calibration-aware learning.

This study investigates the impact of model selection criteria—such as accuracy versus loss—on test performance in neural classifier training, particularly under early stopping with patience. Through systematic empirical evaluation using k-fold cross-validation on standard benchmarks, the work compares multiple validation metrics, including accuracy, cross-entropy, C-Loss, and PolyLoss, under both early stopping and post-hoc full-trajectory selection strategies. The findings reveal that validation loss–based criteria consistently outperform validation accuracy, which exhibits not only inferior performance but also lower stability. More critically, regardless of the selection criterion employed, the chosen models are typically substantially worse than the best test performance observed during training, thereby exposing a fundamental limitation in current model selection paradigms.

This paper addresses the lack of a unified early-stopping mechanism for implicit regularization in iterative learning. To bridge this gap, the authors propose a theory-driven early-stopping framework. They develop EarlyStopping, an open-source Python toolkit that—uniquely—systematically integrates truncated SVD, Landweber iteration, conjugate gradient, L2-boosting, and regression trees. The toolkit supports user-defined data generation and enables real-time monitoring of theoretical regularization strength, including effective degrees of freedom and bias–variance trade-offs. Implemented in NumPy/SciPy, it provides sequential risk estimation and analytically derived stopping boundaries. Experiments reproduce key theoretical results on implicit regularization, demonstrating that principled early stopping effectively suppresses noise propagation, constrains generalization error growth, and significantly enhances algorithmic robustness and interpretability—thereby narrowing the gap between theoretical analysis and practical deployment.

This work addresses the challenge of achieving optimal performance in reasoning models under minimal computational cost by proposing LearnStop, a lightweight, learning-based early-stopping mechanism that operates without access to hidden states. LearnStop dynamically predicts whether the current reasoning prefix is correct by online aggregation of multidimensional features—such as confidence, entropy, and answer stability—at fixed budget points to decide whether to terminate inference early. Theoretical analysis and experiments demonstrate that LearnStop significantly outperforms conventional threshold-based methods specifically in tasks lacking reliable single-scalar signals yet containing early correct answers, such as open-ended mathematical reasoning; on GSM8K, it improves accuracy by 2.8% over the strongest scalar baseline and extends the performance frontier under fixed budgets. However, its advantage is limited in multiple-choice or extremely difficult problems. The study systematically delineates the effectiveness boundary of learned stopping strategies.

Fixed-size benchmarking in model evaluation often fails to balance efficiency, statistical reliability, and diverse objectives, leading to either excessive resource consumption or unreliable results. This work proposes the first adaptive framework that integrates sequential testing into AI model evaluation, dynamically allocating evaluation data based on stopping criteria tailored for model ranking and selection tasks. By combining sequential hypothesis testing, minimum detectable effect analysis, and diminishing returns detection, the method achieves substantial gains in efficiency without compromising rigor. Empirical validation on the Open VLM Leaderboard demonstrates an 80% reduction in computational cost while maintaining a confidence interval width of 2.5 points, significantly enhancing both the practicality and scalability of model evaluation.

This work addresses the challenge of catastrophic forgetting in sequentially trained early-exit neural networks, where newly added exits often disrupt previously learned classifiers, degrading their performance. To mitigate this issue, the authors introduce a stability-plasticity trade-off into early-exit architectures through two complementary mechanisms: first, Elastic Weight Consolidation (EWC) is applied at the parameter level to protect critical weights of existing exits; second, Learning without Forgetting (LwF) is employed at the output level to preserve the output distributions of earlier exits. Evaluated on standard benchmarks, the proposed approach significantly outperforms existing sequential training strategies, achieving higher accuracy and improved inference efficiency with minimal computational overhead, thereby effectively alleviating catastrophic forgetting in sequential early-exit network training.

This work addresses the challenge that large reasoning models often incur excessive computational costs and degraded performance due to over-reasoning, primarily because of the difficulty in determining the optimal stopping point. The study systematically uncovers, for the first time, the distinct dynamic evolution patterns of confidence in intermediate answers between correct and incorrect reasoning trajectories. Building on this insight, the authors propose CoDE-Stop, a plug-and-play early-stopping mechanism that requires no additional training. CoDE-Stop adaptively halts reasoning based on shifts in intermediate answer confidence, achieving a 25%–50% reduction in token consumption across multiple large language models and scientific reasoning benchmarks while maintaining or even improving accuracy—thereby establishing a superior trade-off between accuracy and computational efficiency.

This work addresses the tendency of large language models to generate redundant reasoning when faced with ambiguous or uncertain queries. To mitigate this, the authors propose a statistically grounded early-stopping mechanism that models the arrival time of uncertainty-indicating keywords through a parameterized update process. By integrating sequential hypothesis testing with nonparametric methods, the approach yields an adaptive termination strategy with finite-sample guarantees. Notably, this is the first method to incorporate sequential testing into parameterized early-stopping design, effectively preventing premature termination on well-defined problems while substantially enhancing reasoning efficiency and reliability across multiple domains—particularly excelling in mathematical reasoning tasks.