D$^3$: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training

2026-05-29

AI Summary

This work addresses the common oversight in existing large language model training schedules, which often neglect dynamic inter-sample dependencies and the impact of training order. To tackle this, the authors propose the D³ framework, which introduces a dynamic directed influence graph to model directional relationships among training samples and formulates sequence generation as a graph-constrained optimization problem aligned with information evolution principles. The approach integrates loss-based dynamic graph construction with an efficient approximate solver, balancing theoretical rigor and scalability while maintaining low computational overhead. Experimental results demonstrate that D³ consistently outperforms current scheduling strategies in both pretraining and post-training phases.

Abstract

Training data plays a central role in large language models (LLMs) optimization, motivating extensive research on data scheduling strategies. Most existing approaches concentrate on adjusting the overall data distribution but neglect the underlying interactions between samples during training. However, we argue that such interactions cannot be overlooked, as real-world data samples frequently exhibit directional influences on each other, making the training order crucial. Intuitively, we can prioritize train-units with greater influence to improves learning efficiency. In this work, we propose $D^3$, a Dynamic Directional graph-constrained Data scheduling framework. $D^3$ formulates the complex interactions among train-units as a dynamic influence graph, where edges represent loss-based dependencies. It then solves a constrained optimization problem over this graph to derive the training order, which ensures that the data sequence respects the evolving information flow throughout training. Our approach is theoretically motivated and yields consistent improvements over existing data scheduling methods across both pre-training and post-training phases. Furthermore, for scalability, $D^3$ also employs an efficient approximation algorithm that keeps the additional computational overhead within a manageable range. For future research, the code is available at https://github.com/xuyj233/D3.

Problem

• data scheduling
• training order
• sample interactions
• directional influence
• large language models

Innovation

• Dynamic Influence Graph
• Data Scheduling
• Directional Dependency
• Constrained Optimization
• LLM Training