RailWave: Adaptive Spatial and Temporal Scheduling for Expert-Parallel Communication

2026-10-02

AI Summary

This study addresses the irregular All-to-All communication bottleneck and uneven network rail utilization in expert-parallel Mixture-of-Experts (MoE) models by proposing a DeepEP-based adaptive communication layer. Methodologically, spatial traffic shaping is achieved through source-local information redistribution and reusable topology permutation scheduling, while temporal traffic shaping is accomplished via a lightweight calibration selector. Furthermore, offline profiling techniques are introduced to optimize communication efficiency without reconstructing demand-dependent schedules. Experimental results demonstrate that this approach achieves 5.84× and 4.36× speedups on H800 and H20 platforms, respectively, when applied to the GLM-4.5-Air model, significantly enhancing large-scale MoE training performance.

Abstract

Irregular All-to-All communication is a major bottleneck in expert-parallel Mixture-of-Experts (MoE) models. Even with fixed expert routing and placement, uneven utilization of parallel network Rails and incast can limit communication performance. We present RailWave, a phase-adaptive communication layer built on DeepEP that addresses these bottlenecks below the routing layer through spatial and temporal traffic shaping. RailBalance redistributes source traffic across eligible Rails using source-local information, while a reusable, topology-derived permutation schedule limits concurrent senders per receiver without rebuilding demand-dependent schedules for each communication phase. A lightweight calibrated selector chooses an execution path according to each phase's traffic characteristics and offline profiling results. On training-derived communication workloads from the 106B GLM-4.5-Air model, RailWave delivers up to 5.84x speedup on H800 and 4.36x on H20 over Native. Code is available at https://github.com/CyberSecurityErial/RailWave-EP.

Problem

• Mixture-of-Experts
• All-to-All communication
• expert parallelism
• incast
• network Rails

Innovation

• Expert-Parallel MoE
• All-to-All Communication
• Spatial and Temporal Traffic Shaping
• Topology-derived Permutation Schedule
• Phase-adaptive Scheduling