Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

2026-08-07

AI Summary

This work addresses the systematic bias in traditional neural scaling laws, which assume that model scale and training data size independently affect loss—a simplification that breaks down under data-scarce or overtrained regimes. To overcome this limitation, the authors propose the Skaling law, which explicitly models the coupling between model size and data quantity through a single interaction exponent, yielding a more accurate loss prediction function. By integrating sparse grid experimental design with an efficient extrapolation algorithm, the method achieves 1.5–3× lower mean absolute percentage error in both interpolation and extrapolation tasks, while requiring only approximately one-tenth of the computational cost to match the accuracy of full-grid approaches. This advance significantly improves predictive fidelity and reduces computational overhead in scaling law estimation.

Abstract

Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.

Problem

• neural scaling laws
• language model
• loss estimation
• model-data coupling
• compute efficiency

Innovation

• scaling laws
• model-data coupling
• interaction exponent
• compute-efficient extrapolation
• neural scaling