Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models

2025-06-05

AI Summary

Pretraining data faces a “data wall”: high-quality text is scarce, and conventional filtering discards over 99% of web-crawled data. Method: This paper proposes REWIRE—a novel framework that systematically recovers low-quality, previously filtered web text and transforms it into high-fidelity training samples via large language model–guided controllable rewriting, balancing factual accuracy and lexical/semantic diversity. REWIRE integrates multi-stage quality assessment, hybrid data-ratio optimization, and DCLM—a multi-scale evaluation protocol. Contribution/Results: On 1B/3B/7B models, REWIRE achieves average gains of 1.0–2.5 percentage points across 22 downstream tasks. Using only original + rewritten data, it outperforms models trained on twice the volume of raw web data. Notably, 82% of newly generated tokens originate from originally discarded documents, effectively alleviating both data scale and quality bottlenecks.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Scaling laws predict that the performance of large language models improves with increasing model size and data size. In practice, pre-training has been relying on massive web crawls, using almost all data sources publicly available on the internet so far. However, this pool of natural data does not grow at the same rate as the compute supply. Furthermore, the availability of high-quality texts is even more limited: data filtering pipelines often remove up to 99% of the initial web scrapes to achieve state-of-the-art. To address the"data wall"of pre-training scaling, our work explores ways to transform and recycle data discarded in existing filtering processes. We propose REWIRE, REcycling the Web with guIded REwrite, a method to enrich low-quality documents so that they could become useful for training. This in turn allows us to increase the representation of synthetic data in the final pre-training set. Experiments at 1B, 3B and 7B scales of the DCLM benchmark show that mixing high-quality raw texts and our rewritten texts lead to 1.0, 1.3 and 2.5 percentage points improvement respectively across 22 diverse tasks, compared to training on only filtered web data. Training on the raw-synthetic data mix is also more effective than having access to 2x web data. Through further analysis, we demonstrate that about 82% of the mixed in texts come from transforming lower-quality documents that would otherwise be discarded. REWIRE also outperforms related approaches of generating synthetic data, including Wikipedia-style paraphrasing, question-answer synthesizing and knowledge extraction. These results suggest that recycling web texts holds the potential for being a simple and effective approach for scaling pre-training data.

Problem

• Enhancing pre-training data quality and quantity for language models
• Addressing limited high-quality text availability in web crawls
• Transforming discarded low-quality documents into useful training data

Innovation

• Recycles filtered web data for training
• Enhances low-quality documents via rewriting
• Mixes raw and synthetic texts effectively