ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining

2026-09-28

AI Summary

This study addresses the limitation that reliance on coarse-grained heuristic rules for cleaning large language model (LLM) pretraining data constrains corpus quality. To this end, we propose ReScraper, a framework adopting an AI4AI paradigm that employs a unified 0.6B language model to replace conventional complex rule stacks in an end-to-end manner. Specifically, teacher models generate supervision signals to guide the compact model in collaboratively executing four fine-grained operations: extraction, editing, deletion, and rewriting, thereby substantially enhancing data diversity and quality. Experimental results demonstrate that ReScraper yields relative improvements of 3.8%–4.7% in DCLM Core scores across models of varying scales, significantly outperforming existing state-of-the-art baselines.

Abstract

LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozens of rule-based filters then clean it, so corpus quality is capped by the coarseness and accuracy of the rules. In this work, we propose ReScraper, a unified language model of only 0.6B parameters that replaces this entire stack. To train ReScraper, we carefully curate supervised data from the outputs of three teacher models, so it learns to first extract the main content from raw data and then choose among four operations: keeping the page as extracted, editing out noisy lines and spans, deleting it entirely, or rewriting it when it is poorly written but informative. Based on the same crawled data pool, pretraining 400M, 1.4B, and 2.8B models on our curated data improves the DCLM Core score by a relative 3.8--4.7% over the strongest baseline at each scale, including the costly multi-agent curation. Our analyses show that each operation plays a distinct and complementary role, and that extracting and cleaning in one model outperforms a cascade of separate models. ReScraper also concentrates its operations on the pages that need them, raising the quality of poor pages the most while keeping the corpus diverse. These results demonstrate the feasibility and effectiveness of AI4AI for pretraining data curation, where a small learned model takes over an entire stage of the pipeline from hand-written heuristics. We open-source our code at https://github.com/cxcscmu/ReScraper

Problem

• LLM pretraining
• data curation
• web scraping
• heuristic rules
• corpus quality

Innovation

• Web Data Curation
• Unified Language Model
• LLM Pretraining
• AI4AI
• Data Cleaning