Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

2026-05-11

AI Summary

This work addresses the susceptibility of large language models to noisy web-scale text during pretraining, which can degrade performance. The authors propose a lightweight “pre-pretraining” phase prior to standard pretraining, leveraging synthetic data generated from formal languages—comprising only 65 million tokens—to facilitate learning of temporal structures. This approach is the first to demonstrate that synthetic pre-pretraining effectively suppresses the model’s tendency to self-model noisy inputs and guides attention mechanisms to progressively reduce reliance on corrupted tokens. Experimental results on a 1-billion-parameter model show that this strategy not only matches baseline performance but also reduces the required amount of real-text pretraining tokens by up to 49% across varying noise levels, substantially enhancing both robustness and training efficiency.

Abstract

Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model performance. Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice. We therefore study whether a lightweight pre-pre-training (PPT) stage based on synthetic data with learnable temporal structure helps resist noisy data during the pre-training (PT) stage. Across various corruption settings, our method consistently improves robustness to noise during PT, with larger relative gains at higher noise levels. For a 1B-parameter model, a synthetic PPT stage with only 65M tokens achieves the same final loss as the baseline while using up to 49\% fewer natural-text PT tokens across different noise levels. Mechanistic analyses suggest PPT does not immediately suppress attention to noisy tokens. Rather, PPT-initialized models gradually downweight attention between corrupted tokens during noisy PT. This indicates that synthetic PPT inhibits noise self-modeling and shapes the subsequent optimization trajectory. Code is available at https://github.com/guox18/formal-language-prepretraining.

Problem

• noisy pre-training data
• language model robustness
• pre-training corpora
• data noise
• model degradation

Innovation

• synthetic pre-pre-training
• language model robustness
• noisy pre-training data
• temporal structure
• attention downweighting