Bridging Compute- and Data-Optimal Pretraining

2026-07-28

AI Summary

This work addresses the breakdown of traditional compute-optimal scaling laws in the face of rapidly expanding computational resources outpacing the growth of high-quality pretraining data. The authors propose a Compute-Data (CD) scaling framework that introduces a token effectiveness function, η, to jointly model the interplay among model size, dataset scale, and compute budget, while quantifying the relative effectiveness of derived data—such as duplicates or rewrites—compared to original tokens. Experiments on the Dolma-3 corpus across models ranging from 14M to 600M parameters demonstrate that η is non-constant and exhibits saturation, revealing that classical optimal allocation strategies are often suboptimal in practice. This study further delineates three distinct training regimes—compute-limited, data-limited, and model-limited—establishing a new theoretical foundation for efficient large language model training.

Abstract

Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound. CD scaling extends classical scaling laws by introducing a token-effectiveness function, $η$, which quantifies the value of a derived token-produced, for example, through multi-epoch repetition or paraphrasing-relative to a fresh token, ranging from a perfect substitute to having no value. We fit $η$ for two data-expansion strategies, multi-epoch repetition and paraphrasing, across model sizes from 14M to 600M parameters using the Dolma-3 corpus. We find that token effectiveness is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and it saturates as the corpus is expanded. The functional form of $η$ implies diminishing returns when substituting compute for data as either model size or data availability increases. It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound---and shows that classical compute-optimal allocation is suboptimal across most practically relevant settings.

Problem

• compute-optimal scaling
• data scarcity
• pretraining
• scaling laws
• token effectiveness

Innovation

• scaling laws
• compute-data tradeoff
• token effectiveness
• data-constrained training
• language model pretraining