Mixture-of-Experts Serving

2026-07-20

AI Summary

This work addresses the trade-off among GPU resource allocation, latency, and reconfiguration cost in Mixture-of-Experts (MoE) serving caused by dynamically shifting expert loads. It presents the first formal model for MoE serving and introduces the first polynomial-time online algorithm with a provable competitive ratio of $O(\sqrt{\log k})$, while establishing a matching lower bound of $\Omega(\sqrt{\log k})$ for its dual problem. In the offline setting, the problem is shown to be NP-hard and admit no fully polynomial-time approximation scheme (FPTAS), yet a constant-factor approximation algorithm is provided. By integrating online and offline algorithm design, competitive analysis, and complexity theory under the Exponential Time Hypothesis (ETH), this study lays a theoretical foundation and offers practical solutions for efficient MoE deployment.

Abstract

Mixture-of-Experts (MoE) models route each token to only a few expert networks, distributing the serving load across experts whose popularity shifts over time. A serving system must therefore dynamically decide how many GPUs to assign to each expert, trading off service latency against the cost of reconfiguring the assignment. We introduce a formal model of MoE Serving and initiate a principled study of online and offline algorithms for it. Our main result is a polynomial-time $O(\sqrt{\log k})$-competitive online algorithm, where $k$ is the number of GPUs beyond one per expert. We complement it with a matching $Ω(\sqrt{\log k})$ barrier for the online dual problem underlying our analysis. In the offline setting, we give a constant-factor approximation, show that MoE Serving is NP-hard, and rule out an FPTAS assuming ETH.

Problem

• Mixture-of-Experts
• GPU allocation
• online algorithm
• resource scheduling
• serving latency

Innovation

• Mixture-of-Experts
• online algorithm
• competitive analysis
• resource allocation
• NP-hardness