learning curve analysis

This study investigates the co-optimization of model scale, dataset size, and numerical precision in low-precision training to balance performance and computational cost. Leveraging a high-dimensional sketching-based linear regression framework, the work models quantization error and analyzes theoretical scaling laws to reveal a fundamental distinction between multiplicative and additive quantization: the former preserves the effective capacity of full-precision models, whereas the latter substantially diminishes it. Theoretical analysis characterizes the intricate coupling among model size, data volume, and precision, and extensive experiments confirm markedly different scaling behaviors between the two quantization paradigms. These findings provide a principled foundation and practical design guidelines for efficient low-precision training.

This study addresses how to reliably predict the upper-bound performance of language models on downstream tasks based on pretraining compute budgets and assess the temporal stability of this mapping. Leveraging large-scale empirical observations, the work employs smooth quantile regression to model the monotonic, saturating S-shaped relationship between pretraining FLOPs and high-percentile downstream performance, establishing the first stable and predictable performance frontier. Key contributions include uncovering the continually advancing frontier of mathematical reasoning capabilities, proposing an efficient algorithm that approximates the full performance frontier using only 20% of the evaluation budget, and releasing the Proteus-2k benchmark dataset. Empirical validation demonstrates strong temporal stability of capability frontiers across most tasks—except mathematical reasoning—substantially reducing evaluation costs.

Traditional scaling laws rely solely on model and data scale to predict performance, neglecting other hyperparameters and thus struggling to achieve accurate prediction and efficient tuning under hardware constraints. This work proposes Configuration-to-Performance Scaling Laws (CPL), which, for the first time, incorporate the full training configuration into the modeling framework. By parameterizing this mapping with a large language model, the authors introduce a neuralized CPL (NCPL). Trained on open-source pretraining logs, NCPL enables joint optimization across multiple hyperparameters and predicts loss curves with 20–40% lower error than Chinchilla scaling laws. It generalizes effectively to regimes up to ten times the maximum compute budget observed in the training set and matches baseline methods in multi-hyperparameter tuning tasks.

This work investigates whether pretraining metrics—such as perplexity—reliably predict downstream performance of large language models (LLMs) after fine-tuning, aiming to improve model selection efficiency under fixed computational budgets. The authors formulate checkpoint selection as a pairwise classification task and systematically evaluate 50 distinct 1B-parameter LLM variants across diverse downstream tasks, revealing that perplexity is frequently misleading. They propose novel unsupervised and supervised proxy metrics, which reduce prediction error rates by over 50% in multi-task supervised fine-tuning (SFT) evaluation. This study is the first to empirically demonstrate a non-monotonic relationship between pretraining metrics and downstream performance. The proposed proxies exhibit strong cross-task generalization and practical utility, offering a trustworthy, task-aware evaluation paradigm for optimizing pretraining strategies toward downstream objectives.

This work addresses the poor robustness and low predictive accuracy of machine learning model scaling laws. We propose a systematic, reproducible framework for scaling law modeling and evaluation. Based on large-scale empirical analysis across 485 pretrained models on downstream tasks, we (i) demonstrate for the first time that leveraging intermediate training checkpoints significantly improves fitting accuracy; (ii) find that parameter transfer between isomorphic models outperforms cross-size extrapolation; and (iii) verify that averaging estimates from multiple small models trained with different random seeds is more robust than relying on a single large model. We release the first open-source, multidimensional scaling law benchmark dataset, integrating log-log linear regression, training trajectory analysis, and statistical robustness assessment. Our framework reduces average prediction error by 37% across diverse architectures, providing efficient and reliable quantitative guidance for key pretraining decisions—including optimizer selection, dataset curation, and architectural design.

Traditional scaling law estimation suffers from high computational costs due to the absence of efficient budget allocation strategies. This work proposes a novel approach that, for the first time, integrates surrogate-guided pruning into scaling law modeling by combining the Successive Halving algorithm with both parametric and non-parametric surrogate models. This integration enables proactive allocation of computational resources and efficient construction of loss-compute Pareto frontiers. The method substantially improves resource utilization efficiency, achieving relative performance gains of up to 2.84% on real datasets and 5.47% on synthetic datasets, while reducing computational costs by as much as 98.7%.

This work addresses the performance degradation of time series models in post-training quantization (PTQ), which arises from error propagation and amplification during quantization—particularly challenging in calibration-free or black-box settings where module sensitivity is hard to assess. To tackle this, the paper introduces discrete-time dynamical systems theory into quantization analysis for the first time. By modeling the inference process as a dynamical system, it proposes TQS, a quantizer-agnostic, prior-based sensitivity metric derived from trajectory sensitivity analysis, enabling calibration-free mixed-precision quantization budget allocation. The resulting TQS-PTQ framework significantly outperforms existing PTQ methods without relying on calibration data or second-order approximations, facilitating efficient low-bit deployment.

This work addresses the critical challenge of dynamically determining when to perform continual fine-tuning of foundation models on resource-constrained devices under limited computational budgets to maximize performance. The problem is formally cast, for the first time, as a constrained Markov decision process, where the state encompasses model performance, remaining compute budget, and the relevance of incoming data to the historical distribution. The authors propose an online decision-making strategy based on an Actor-Critic reinforcement learning framework; when fine-tuning gains are predictable, dynamic programming is also integrated for optimal scheduling. Experimental results demonstrate that the proposed approach improves accuracy by over 4% compared to strong baselines under identical budgets and achieves 97% of the performance of full-parameter fine-tuning using only 25% of the fine-tuning steps.

This work addresses the semantic gap between evaluation metrics and training data in large model pretraining, which hinders precise diagnosis and remediation of capability deficiencies. The authors propose “capability slices” as fundamental units aligning evaluation and data, establishing a bidirectional classification framework that links evaluation tasks with non-instructional training data through explicit mapping rules. This enables a closed-loop pipeline from evaluation failures to targeted data interventions. For the first time, the approach supports auditable and systematic reasoning that translates evaluation signals into data corrections, moving beyond intuition-driven tuning paradigms. Experiments demonstrate its efficacy in both directions: repairing specific training loss components restores BBH performance to 66.44, while targeted data sampling boosts AIME2025/2026 Pass@128 from 6.67/0.00 to 26.67.

This work addresses the high cost of large model fine-tuning by tackling the challenge of accurately predicting post-fine-tuning performance beforehand—a task whose theoretical limits remain unclear. We formulate pre-fine-tuning performance prediction as a stochastic estimation problem under information constraints and introduce a predictive risk decomposition framework that separates it into an irreducible intrinsic limit and an optimizable variance term, thereby revealing fundamental bounds on predictability. Leveraging information theory and statistical learning theory, we establish a theoretical lower bound on variance decay through optimization and construct a predictability phase diagram that delineates three distinct task regimes. Experiments on both synthetic and real-world benchmarks validate the efficacy of this phase diagram, and our proposed budget-optimal probing strategy significantly enhances prediction efficiency, offering both theoretical grounding and practical tools for pre-fine-tuning decision-making.