A Hitchhiker's Guide to Scaling Law Estimation

2024-10-15

arXiv.org

AI Summary

This work addresses the poor robustness and low predictive accuracy of machine learning model scaling laws. We propose a systematic, reproducible framework for scaling law modeling and evaluation. Based on large-scale empirical analysis across 485 pretrained models on downstream tasks, we (i) demonstrate for the first time that leveraging intermediate training checkpoints significantly improves fitting accuracy; (ii) find that parameter transfer between isomorphic models outperforms cross-size extrapolation; and (iii) verify that averaging estimates from multiple small models trained with different random seeds is more robust than relying on a single large model. We release the first open-source, multidimensional scaling law benchmark dataset, integrating log-log linear regression, training trajectory analysis, and statistical robustness assessment. Our framework reduces average prediction error by 37% across diverse architectures, providing efficient and reliable quantitative guidance for key pretraining decisions—including optimizer selection, dataset curation, and architectural design.

Technology Category

• Machine Learning: Scalability of ML Systems
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets. This provides an efficient way for practitioners and researchers alike to compare pretraining decisions involving optimizers, datasets, and model architectures. Despite the widespread use of scaling laws to model the dynamics of language model training, there has been little work on understanding how to best estimate and interpret them. We collect (and release) a large-scale dataset containing losses and downstream evaluations for 485 previously published pretrained models. We use these to estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families. We find that fitting scaling laws to intermediate checkpoints of training runs (and not just their final losses) substantially improves accuracy, and that -- all else equal -- estimates of performance are generally most accurate when derived from other models of similar sizes. However, because there is a significant degree of variability across model seeds, training multiple small models is sometimes more useful than training a single large one. Moreover, while different model families differ scaling behavior, they are often similar enough that a target model's behavior can be predicted from a single model with the same architecture, along with scaling parameter estimates derived from other model families.

Problem

• Estimating scaling laws for machine learning model loss prediction
• Improving accuracy by using intermediate training checkpoints
• Predicting target model behavior from similar architectures

Innovation

• Use intermediate checkpoints for better accuracy
• Compare models of similar sizes for estimates
• Predict behavior from single similar architecture