scaling law analysis

Traditional neural scaling laws exhibit diminishing applicability in emerging architectures—including sparse models, Mixture-of-Experts (MoE), multimodal systems, and retrieval-augmented models—due to heterogeneity across modalities and stringent deployment constraints. Method: This work systematically reviews the theoretical foundations and empirical boundaries of scaling laws, synthesizing insights from over 50 studies. We propose an adaptive scaling framework that jointly optimizes data efficiency, inference cost, and architectural constraints, integrating power-law modeling, cross-modal performance attribution, and architecture-sensitivity analysis. Contribution/Results: We distill transferable, practical scaling guidelines that precisely delineate the validity domains and failure modes of scaling laws. The framework provides principled theoretical support and actionable decision-making tools for efficient large-model scaling; its components have been adopted by multiple industrial-scale training frameworks.

Traditional scaling law estimation suffers from high computational costs due to the absence of efficient budget allocation strategies. This work proposes a novel approach that, for the first time, integrates surrogate-guided pruning into scaling law modeling by combining the Successive Halving algorithm with both parametric and non-parametric surrogate models. This integration enables proactive allocation of computational resources and efficient construction of loss-compute Pareto frontiers. The method substantially improves resource utilization efficiency, achieving relative performance gains of up to 2.84% on real datasets and 5.47% on synthetic datasets, while reducing computational costs by as much as 98.7%.

Traditional scaling laws rely solely on model and data scale to predict performance, neglecting other hyperparameters and thus struggling to achieve accurate prediction and efficient tuning under hardware constraints. This work proposes Configuration-to-Performance Scaling Laws (CPL), which, for the first time, incorporate the full training configuration into the modeling framework. By parameterizing this mapping with a large language model, the authors introduce a neuralized CPL (NCPL). Trained on open-source pretraining logs, NCPL enables joint optimization across multiple hyperparameters and predicts loss curves with 20–40% lower error than Chinchilla scaling laws. It generalizes effectively to regimes up to ten times the maximum compute budget observed in the training set and matches baseline methods in multi-hyperparameter tuning tasks.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

This study systematically investigates the scaling laws of data-driven weather forecasting models, elucidating how model size, training data volume, and computational budget jointly influence predictive performance. Through large-scale empirical analysis comparing state-of-the-art models such as Aurora and GraphCast under diverse configurations, the work reveals that weather models exhibit scaling behaviors markedly distinct from those of language models: performance gains are more sensitive to increasing network width than depth, and extended training time yields greater improvements than simply scaling up model parameters. Experiments demonstrate that Aurora achieves the highest data-scaling efficiency—yielding a 3.2× reduction in loss with a 10× increase in data—while GraphCast exhibits superior parameter efficiency. The study further proposes optimized strategies for allocating computational resources, offering both theoretical insights and practical guidance for efficient weather modeling.

This work addresses the poor robustness and low predictive accuracy of machine learning model scaling laws. We propose a systematic, reproducible framework for scaling law modeling and evaluation. Based on large-scale empirical analysis across 485 pretrained models on downstream tasks, we (i) demonstrate for the first time that leveraging intermediate training checkpoints significantly improves fitting accuracy; (ii) find that parameter transfer between isomorphic models outperforms cross-size extrapolation; and (iii) verify that averaging estimates from multiple small models trained with different random seeds is more robust than relying on a single large model. We release the first open-source, multidimensional scaling law benchmark dataset, integrating log-log linear regression, training trajectory analysis, and statistical robustness assessment. Our framework reduces average prediction error by 37% across diverse architectures, providing efficient and reliable quantitative guidance for key pretraining decisions—including optimizer selection, dataset curation, and architectural design.

This work addresses the challenges of SLO violations and resource inefficiency in machine learning model serving caused by inadequate capacity planning. To this end, the authors propose an adaptive, feedback-driven load testing framework that formalizes the ML serving load testing process for the first time. The framework incorporates real-traffic-based workload calibration and a warm-up mechanism, combined with adaptive search, performance signal feedback control, convergence detection, and GPU monitoring to efficiently estimate the maximum sustainable throughput under SLO constraints. Evaluation across 14 industrial cases demonstrates that the approach reduces capacity estimation error from approximately 30% to 2–6%, with the warm-up mechanism improving accuracy by 22.2%. This significantly mitigates deployment incidents and enhances GPU resource utilization efficiency.

This work addresses the strong dependence of traditional models on parameter scale in high-energy physics by introducing, for the first time in particle physics, an active control strategy over neural scaling laws for hadronic jet classification. By leveraging a high-fidelity simulator to generate low-cost synthetic data, the authors reconstruct the pretraining data distribution to better align with the downstream task while enhancing its diversity. This approach shifts the performance scaling paradigm from reliance on model size to dependence on data volume, yielding significantly improved classification accuracy under identical computational budgets. The study thus demonstrates effective engineering control over scaling behavior through deliberate manipulation of pretraining data composition.

This work addresses the common misconception that scaling laws apply only to large models, which often arises because small-scale models are evaluated with suboptimal hyperparameters. Through systematic analysis, the study demonstrates that scaling laws remain valid even for small models when evaluated along a properly tuned hyperparameter frontier, and further reveals that hyperparameter sensitivity diminishes as model scale increases. Building on these insights, the authors propose a new paradigm that combines small-scale experiments with efficient hyperparameter optimization. Using ablation studies, loss landscape analysis, and scaling modeling, they successfully reproduce findings typically observed only at large scales—such as the superiority of pre-normalization—thereby establishing that, under appropriate methodology, small-scale experiments can reliably predict large-model behavior.