Scaling Native Multimodal Pre-Training From Scratch

2026-07-24

AI Summary

This work addresses the limitations of current large language models, which struggle to perceive the multimodal physical world due to their reliance solely on text-based pretraining, and investigates the unclear scaling laws governing natively multimodal pretraining. Under a fixed compute budget, the study systematically examines the optimal allocation between model size and training token count in Transformer-based vision–language models, analyzing how data composition affects scaling behavior across both multimodal and language tasks. Through from-scratch pretraining, power-law fitting, and cross-modal transfer analyses, the authors uncover fundamental differences in scaling dynamics between language-only and multimodal objectives, propose a data-composition-driven rule for compute allocation, and establish a compute-efficient frontier. Experiments demonstrate that optimal loss follows predictable scaling laws and that multimodal pretraining substantially enhances both pure-text reasoning and multimodal in-context learning capabilities.

Abstract

Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.

Problem

• multimodal pre-training
• scaling laws
• compute-optimal
• data composition
• vision-language models

Innovation

• native multimodal pre-training
• scaling laws
• compute-optimal allocation
• cross-modal transfer
• efficiency frontier