Zepp: Accelerating Distributed MoE Serving under Relaxed Balance Constraints

2026-10-07

AI Summary

This study addresses the computational, communication, and memory imbalances caused by skewed expert loads in distributed Mixture-of-Experts (MoE) serving. We propose a system framework that formulates load balancing as a hard constraint rather than an optimization objective. The core innovation lies in introducing the concept of relaxed balance constraints, which jointly optimizes expert placement, token routing, and execution scheduling to co-adapt to dynamic workloads. Furthermore, we design Split/Merge primitives alongside communication overlapping mechanisms to enable efficient expert parallelism under GPU and NIC resource constraints. Experimental results demonstrate that the proposed system achieves up to 6.68× speedup over seven state-of-the-art baselines, with a geometric mean speedup of 1.86×.

Abstract

As Mixture-of-Experts (MoE) models continue to scale, serving them increasingly relies on expert parallelism (EP) across a growing number of devices. Yet skewed expert workloads create imbalance across computation, communication, and memory, making load balancing a central optimization objective in distributed MoE serving. We observe that balance is not free: operations introduced to balance one dimension can themselves be expensive or imbalanced. This motivates us to rethink balance as a constraint rather than an optimization objective. We present Zepp, which directly optimizes the bottleneck communication in distributed MoE serving subject to simplified balance constraints on physical resources, i.e., GPUs and NICs. Zepp progressively optimizes inter-node communication across placement, routing, and execution. It first places expert replicas to reduce token communication under GPU constraints, then reshapes communication flows through split and merge primitives under NIC constraints, and finally partitions and schedules ex- pert computation to overlap the resulting communication. To adapt to dynamic workloads, Zepp jointly coordinates computation, token communication, and expert-weight movement at each iteration. Together, these designs allow Zepp to pursue the most efficient execution rather than a single-dimension balanced one. We implement Zepp and evaluate it against 7 state-of-the-art MoE serving systems, achieving up to 6.68$\times$ MoE layer speedup and a geometric mean speedup of 1.86$\times$ over the fastest competing baseline.

Problem

• Mixture-of-Experts
• Distributed Serving
• Load Balancing
• Expert Parallelism
• Bottleneck Communication

Innovation

• Mixture-of-Experts
• Distributed Serving
• Load Balancing
• Communication Optimization
• Expert Parallelism