Temporal Scaling Law for Large Language Models

2024-04-27

arXiv.org

AI Summary

This work addresses the lack of understanding regarding the dynamic evolution of test loss during large language model (LLM) pretraining. We propose the **Temporal Scaling Law**, the first scaling law modeling fine-grained loss dynamics at the **token-position level**. Methodologically, we design a dynamic hyperbolic loss function, derive a learnable parametric form governed by temporal power laws, and achieve high-accuracy modeling via token-level loss analysis and parameter evolution inference. Experiments demonstrate that the law enables precise cross-step test loss prediction on both in-distribution and out-of-distribution data (mean error < 0.5%) and supports direct, performance-targeted hyperparameter optimization—e.g., data mixture ratios—significantly improving pretraining efficiency and interpretability. Our core contribution is advancing scaling laws from model-level to token-level granularity, establishing the first dynamic, fine-grained, and generalizable temporal framework for pretraining loss modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recently, Large Language Models (LLMs) have been widely adopted in a wide range of tasks, leading to increasing attention towards the research on how scaling LLMs affects their performance. Existing works, termed Scaling Laws, have discovered that the final test loss of LLMs scales as power-laws with model size, computational budget, and dataset size. However, the temporal change of the test loss of an LLM throughout its pre-training process remains unexplored, though it is valuable in many aspects, such as selecting better hyperparameters extit{directly} on the target LLM. In this paper, we propose the novel concept of Temporal Scaling Law, studying how the test loss of an LLM evolves as the training steps scale up. In contrast to modeling the test loss as a whole in a coarse-grained manner, we break it down and dive into the fine-grained test loss of each token position, and further develop a dynamic hyperbolic-law. Afterwards, we derive the much more precise temporal scaling law by studying the temporal patterns of the parameters in the dynamic hyperbolic-law. Results on both in-distribution (ID) and out-of-distribution (OOD) validation datasets demonstrate that our temporal scaling law accurately predicts the test loss of LLMs across training steps. Our temporal scaling law has broad practical applications. First, it enables direct and efficient hyperparameter selection on the target LLM, such as data mixture proportions. Secondly, viewing the LLM pre-training dynamics from the token position granularity provides some insights to enhance the understanding of LLM pre-training.

Problem

• Exploring how LLM test loss changes with training steps
• Developing fine-grained temporal scaling law for token positions
• Enabling direct hyperparameter selection via loss prediction

Innovation

• Proposes Temporal Scaling Law for LLMs
• Analyzes fine-grained token position loss
• Derives dynamic hyperbolic-law parameters