HyDra: Demystifying and Taming Dynamic Context Parallelism at Production Scale

2026-09-28

AI Summary

This study addresses the load imbalance and pipeline bubble issues arising from dynamic parallelism in large-scale long-context training by proposing a joint computation-communication optimization framework. Methodologically, it designs a closed-loop workload-targeted scheduler coupled with a lazy heap sequence placement strategy to achieve balanced scheduling. Furthermore, it constructs a Ulysses-nested ring architecture to support workload-driven dynamic parallelism and introduces an efficient memory management mechanism to eliminate training bottlenecks. Extensive experiments conducted on thousand-GPU clusters demonstrate that the proposed approach yields an average 1.25× throughput improvement, significantly reduces pipeline bubbles, and accelerates the scheduling process.

Abstract

Long-context training runs on sequences whose lengths span orders of magnitude, and dynamic context parallelism (DCP) gives each sequence its own CP degree. Existing DCP systems either do not scale or perform poorly on mainstream models, leaving Megatron-Core (Mcore) DCP as the only option at production scale. Mcore DCP, however, sizes each degree to fit memory, which grows linearly with length while attention grows quadratically, so comparable token counts hide unequal computation. In our production 256K-context training job on more than 11K GPUs under Mcore DCP, per-rank microbatch times differ by up to 5x at similar token counts. The skew leads to a 46% pipeline bubble and a 13% data-parallel bubble. We present HyDra, a scalable load-driven DCP system. Its scheduler balances computation by pulling every rank toward one load target, and balance in turn makes that target solvable in closed form. It places sequences with lazy heaps rather than whole-pool scans. That balance asks for CP degrees larger than memory requires, so its CP engine nests an inner Ulysses group in a shallow outer ring, letting a higher degree lower computation and communication together. Evaluation at both scales shows consistent gains. On a 512-GPU testbed, HyDra raises throughput over Mcore DCP by 1.18x on average at 32K context and 2.48x at 256K. On a 2,048-GPU production job, it shrinks the pipeline bubble from 36% to 14%, cuts scheduling time by 2.3x, and raises throughput by 1.10-1.43x (avg. 1.25x) over Mcore DCP and 1.33-1.90x (avg. 1.59x) over static CP.

Problem

• Dynamic Context Parallelism
• Long-context Training
• Load Imbalance
• Pipeline Bubble
• Distributed Training

Innovation

• Dynamic Context Parallelism
• Load Balancing
• Long-context Training
• Nested Parallelism
• Pipeline Bubble