Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training

2026-07-30

AI Summary

This work addresses the prevailing limitation in synthetic data generation, which has largely focused on content creation or local rewriting while neglecting the impact of book-level structure on language model training. The authors propose a scalable synthesis pipeline that leverages topic clustering, hierarchical outline planning, and paragraph-aligned generation to construct 686K structured synthetic textbooks comprising 32 billion tokens. Through controlled ablation experiments—the first of their kind—they demonstrate that training on data with coherent book-like organization yields significantly better model performance compared to baselines such as unstructured splitting (Split), random concatenation (RandomConcat), or simple rephrasing (Rephrase). The structured data consistently improves results across multiple downstream tasks, achieving an average gain of +1.09, thereby revealing the critical role of document organization in enhancing training data quality.

Abstract

Synthetic textbook data has improved language model pre-training, but prior work largely treats the benefit as a property of generated content or local rewriting style. We study a different factor: whether related content is organized into coherent book-level documents. We contribute both a scalable synthesis pipeline and controlled evidence that this organization matters. The pipeline retrieves source material from a pre-training corpus, clusters it into topical units, plans hierarchical tables of contents, and assembles source-grounded sections into complete books (our Full setting), yielding 686K textbooks (32B tokens) across 15,000+ disciplines. Replacing natural books in a mid-training mix with this corpus improves downstream performance by +1.09 on average. Controlled comparisons then disentangle the relevant design factors. A content-matched Split condition holds generated text and tokens fixed but treats each section as an independent document; Full's +1.02 mean gain isolates document packaging. A length-matched RandomConcat control that joins sections from different books remains below Full, ruling out document length alone. A retrieval-pool-matched Rephrase condition independently rewrites individual retrieved documents under the same audience-by-style scheme, without clustering, TOC planning, or book assembly; Full's +1.17 gain demonstrates the value of structured synthesis. On Llama3-8B, Full likewise outperforms both RandomConcat and Natural Books, supporting book-level organization as a useful axis for synthetic pre-training data design.

Problem

• synthetic textbook data
• book-level organization
• language model pre-training
• document coherence
• mid-training

Innovation

• book-level organization
• synthetic textbook data
• structured synthesis
• mid-training
• hierarchical table of contents