SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models

2026-06-24

AI Summary

This work addresses the inference bottleneck in sparse Mixture-of-Experts (MoE) models on memory-constrained devices, where expert offloading induces routing-dependent communication overhead that forces sequential execution. The authors propose SpecPrefetch, a novel framework that decouples expert prefetching from routing decisions for the first time: a lightweight shared adapter asynchronously predicts candidate experts for the next layer to enable proactive fetching, while the original frozen router retains authority over final expert selection, ensuring prediction errors affect only transfer efficiency—not output correctness. Integrated with a window-aware scheduler that prioritizes transfers under bandwidth and cache constraints, SpecPrefetch achieves the highest average expert recall in 9 out of 10 configurations across Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny, uses significantly fewer trainable parameters than baselines, and delivers up to a 20% inference throughput improvement on the Snapdragon 8 Elite platform.

Abstract

Sparse Mixture-of-Experts (MoE) models expand foundation model capacity through conditional expert activation, but their full expert pools remain difficult to deploy under limited accelerator memory. Although expert offloading alleviates memory pressure by moving inactive experts to host memory or storage, it introduces a routing-dependent transfer bottleneck: required experts are known only after native top-\(K\) routing, which serializes routing, expert loading, and expert execution during inference. To address this bottleneck, we propose SpecPrefetch, a parameter-efficient prefetching framework for offloaded MoE inference. SpecPrefetch uses a shared lightweight adapter to predict next-layer expert candidates only for asynchronous transfer, while the frozen native router still determines the final executed experts. By separating transfer prediction from execution routing, SpecPrefetch reduces exposed expert-loading latency without changing pretrained routing semantics, so prediction errors affect transfer efficiency rather than model outputs. In addition, a window-aware scheduler prioritizes feasible transfers under cache and bandwidth constraints. Across Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny, SpecPrefetch achieves the best average expert recall in 9 out of 10 model-benchmark settings with substantially fewer trainable parameters than learned predictor baselines. On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to \(20\%\) over a compute-optimized offloading runtime, demonstrating practical benefits for storage-constrained MoE deployment. The code and model weights are available at https://github.com/wei390/SpecPrefetch.

Problem

• Sparse Mixture-of-Experts
• expert offloading
• routing bottleneck
• memory-constrained deployment
• inference latency

Innovation

• SpecPrefetch
• Sparse MoE
• expert prefetching
• parameter-efficient
• offloading