Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

2026-06-09

AI Summary

This work addresses the high computational cost of existing pretraining data selection methods, which often rely on auxiliary models or labeled data. The authors propose WebGraphMix, a novel framework that leverages the host-level web graph topology derived from Common Crawl to guide data mixing without requiring additional training or annotations. By employing unsupervised centrality scores, WebGraphMix adaptively balances the proportion of core and peripheral documents, revealing their complementary learning value in the web graph—a signal orthogonal to existing content-quality metrics. Evaluated on the DataComp-LM benchmark, a simple 1:1 mixing strategy achieves an average score of 41.4% on 400M and 1B language models, outperforming uniform sampling (39.8%). Further gains are realized by combining this approach with content-quality scoring, yielding a performance of 43.8%.

Abstract

The performance of modern language models depends critically on pretraining data composition. Yet existing data selection methods rely on auxiliary classifiers for document scoring or mixture optimization, adding computational overhead and dependence on labeled data. We propose WebGraphMix, a lightweight data selection framework that computes structural centrality scores over the Common Crawl host-level web graph and uses them to vary the proportion of central versus peripheral documents in the pretraining mixture. We hypothesize that central hosts expose models to reusable abstractions, while peripheral hosts encode specialized, long-tail knowledge. WebGraphMix computes centrality scores efficiently at web scale, requiring no model training, labeled data, or downstream supervision. We integrate WebGraphMix into the DataComp-LM pipeline and train models at 400M and 1B parameter scales with 8B and 28B tokens respectively, evaluating on 23 tasks ranging from factual knowledge to symbolic reasoning. Our experiments show that central and peripheral web regions encode complementary capabilities. Mixture combining both at a ratio of 1:1 achieves 41.4% on average, compared to 39.8% for uniform sampling. Combining structural scores with document-level quality classifier scores further improves performance to 43.8%. These findings demonstrate that web graph topology is a meaningful axis for pretraining data curation, capturing information that is largely orthogonal to existing content-based approaches.

Problem

• pretraining data selection
• web graph centrality
• language models
• data curation
• Common Crawl

Innovation

• web graph centrality
• pretraining data selection
• WebGraphMix
• data curation
• language model pretraining