ScAn-Bench: Evaluating Scaling Analysis Methodology

2026-09-28

AI Summary

This study addresses the lack of systematic evaluation regarding data acquisition and extrapolation methodologies in current scaling law research for large models. To this end, we construct a proxy benchmark encompassing both large language models (LLMs) and vision-language models (VLMs), conducting extensive cross-modal experiments across numerous checkpoints to systematically assess the performance of diverse data acquisition strategies and scaling law extrapolation techniques. Our primary contribution is the establishment of the first standardized evaluation framework for cross-modal scaling analysis methodologies, alongside the release of the ScAn-Bench benchmark. By effectively bridging the gap in methodological assessment within scaling analysis, this work provides a reliable evaluation infrastructure to facilitate future research on large model scaling.

Abstract

Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 4524 and 8024 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.

Problem

• Scaling Laws
• Scaling Analysis
• Foundation Models
• Methodology Evaluation

Innovation

• Scaling Laws
• Surrogate Benchmarks
• Extrapolation Methodology
• Foundation Models
• Scaling Analysis