Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation

2026-09-18

AI Summary

研究通过改进Mixture-of-Experts (MoE)路由设计，结合Speculative Decoding提高模型推理速度，特定设计选择显著增强了专家协同激活，提高了吞吐量。

Abstract

Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inference speed of dense models by verifying groups of tokens in parallel. However, the inference speedup for SD with MoEs depends heavily on the number of tokens being verified. Using more verification tokens results in more experts being transferred from DRAM to the Neural Processing Unit (NPU), which increases the memory transfer cost. This negatively impacts model runtime, as memory transfer is typically the bottleneck in inference. In this work, we investigate the impact of MoE router design during training on the speed of MoEs with SD. We find that routers with high degrees of expert coactivation result in much faster runtimes, mitigating the impact of using more verification tokens. Motivated by this observation, we assess the impact of various router design choices on expert coactivation and runtime using billion-parameter transformer models. We find that combining a global load-balancing loss, shared experts, a consistency loss, and an autoregressive expert selection mechanism during training results in significantly stronger expert coactivation. This increased coactivation translates into higher overall runtime throughput: our exploration yields a model that improves throughput by 21% over MoE baselines, while maintaining on-par accuracy with the baseline MoE.

Problem

• Mixture-of-Experts
• Speculative Decoding
• inference speed
• memory transfer cost
• expert coactivation

Innovation

• Mixture-of-Experts
• Speculative Decoding
• Expert Coactivation
• Load-Balancing Loss
• Consistency Loss