DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

2026-07-27

AI Summary

This work proposes DataOrchestra, a novel framework that addresses the limitations of conventional pretraining data processing, which typically applies uniform strategies and fails to account for sample-level heterogeneity. DataOrchestra introduces the first automated, per-sample orchestration mechanism, featuring a hierarchical decision architecture: a high-level orchestrator dynamically selects among discarding, retaining, or combining multiple cleaning operations—such as programmatic editing and large language model–based rewriting—while low-level tool models execute the chosen operations accordingly. Evaluated across 11 benchmark tasks on models ranging from 0.5B to 7B parameters, this approach consistently outperforms any single static processing strategy, demonstrating particularly strong gains in mathematical continual pretraining while substantially reducing computational overhead.

Abstract

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

Problem

• pretraining data curation
• per-example processing
• data cleaning
• large language models
• adaptive data processing

Innovation

• DataOrchestra
• per-example curation
• adaptive data processing
• orchestrated pipeline
• LLM-based rewriting