EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

2026-08-05

AI Summary

This work addresses the memory access bottleneck arising from the combination of Mixture-of-Experts (MoE) and speculative decoding in large language models deployed on edge devices. To overcome this challenge, the authors propose EdgeXpert, a hardware-software co-designed accelerator that efficiently integrates MoE with speculative decoding for the first time. The design introduces a prompt-level expert reuse strategy during the prefill phase and a depth-aware expert fusion mechanism during decoding, significantly reducing memory traffic while preserving model accuracy. Implemented in Samsung’s 28nm process and operating at 800 MHz, EdgeXpert achieves a 56.3% reduction in system latency and a 44.1% decrease in energy consumption, with accuracy closely matching that of the baseline model.

Abstract

On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixture-of-experts (MoE) are promising solutions. Speculative decoding reduces the number of decoding stages by generating multiple tokens per stage, and MoE minimizes per-stage cost through sparse expert activation. However, there is an incompatibility when combining these two techniques. We propose EdgeXpert, a software-hardware co-designed LLM accelerator that resolves this incompatibility. In the prefill stage, the prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than independent per-token expert selection. It identifies important tokens using a lightweight encoder, constructs a shared expert set from them, and routes less important tokens with a reduced expert budget to lower expert EMA. In the decode stage, depth-aware expert coalescing exploits the contextual similarity and mutual exclusivity of same-depth candidate tokens. Rather than loading the union of all required channels, EdgeXpert loads only salient channels and applies computational calibration to recover accuracy without additional memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.

Problem

• Large Language Models
• Edge Inference
• External Memory Access
• Mixture-of-Experts
• Speculative Decoding

Innovation

• Mixture-of-Experts
• Speculative Decoding
• Edge LLM Inference
• Memory-Efficient Acceleration
• Expert Reuse