AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding

2026-08-03

AI Summary

This work addresses the high computational, memory, and communication overhead in speculative decoding for Mixture-of-Experts (MoE) models, which stems from parallel activation of numerous experts and reliance on manually specified expert budgets. The authors propose a verifier-side adaptive expert selection mechanism that dynamically adjusts the number of activated experts per layer by integrating target router scores with offline-estimated commitment probabilities. Notably, they introduce cache residency as a novel criterion to guide expert eligibility under offloading scenarios. The method eliminates the need for predefined expert budgets, reduces host-to-device data transfers by 73.6%–77.1%, and achieves only a 0.27 percentage point average accuracy drop across twelve model-task combinations while improving throughput to 1.29× (fully resident) and 2.06× (offloaded) over the baseline.

Abstract

Speculative decoding verifies a tree of draft tokens in one target-model forward pass. For a mixture-of-experts (MoE) target, however, parallel verification can activate the union of the experts selected by all tree nodes, even though only a small subset of those nodes reaches the accepted output. Token count, activated-expert union size, and expert-weight traffic are therefore distinct cost measures: reducing the token workload need not shrink the expert union proportionally, and under offloading, transfer traffic also depends on cache residency. We introduce AcceptMoE, a verifier-side expert selector that combines target-router scores with offline-estimated commitment probabilities and automatically adjusts the number of eligible experts for each verification block, eliminating the need for a user-specified expert budget. Under offloading, AcceptMoE conditions expert eligibility on cache residency instead of predicting natural routes and prefetching the corresponding expert weights. Although constraining target-expert eligibility changes the model distribution, across 12 model-task pairs spanning three MoE targets and four benchmarks, AcceptMoE's mean accuracy is 0.27 percentage points lower than that of EAGLE-3 speculative decoding with natural routing. Served with SGLang at batch size one, it reaches 1.290 times the throughput of this baseline with all expert weights in GPU memory, and 2.06 times under physical expert offloading, while reducing host-to-device traffic by 73.6 percent to 77.1 percent.

Problem

• Mixture-of-Experts
• Speculative Decoding
• Expert Selection
• Model Offloading
• Verification Efficiency

Innovation

• AcceptMoE
• speculative decoding
• mixture-of-experts
• expert offloading
• commitment-weighted selection