PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

2026-07-17

AI Summary

This work addresses the challenge in MoE-based large language models where expert weights and KV cache contend for limited GPU memory during KV cache-intensive inference. To tackle this, we propose PagedWeight, the first dynamic, quality-aware weight paging and quantization mechanism tailored for MoE architectures. PagedWeight intelligently adjusts the precision of expert weights at runtime based on memory pressure, thereby optimizing the trade-off between memory usage, throughput, and task quality. Experimental results demonstrate that PagedWeight reduces GPU memory consumption by up to 72.0% while preserving FP16-level accuracy and achieves a 1.94× throughput improvement. Under a fixed memory budget, it outperforms existing quantization methods by up to 39.3% in task quality with no more than a 4.1% throughput overhead.

Abstract

Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model's weights at runtime and balances expert-weight precision with the KV cache sizes. PagedWeight exposes and effectively navigates the complex tradeoff between the model's task accuracy, memory consumption, and throughput/latency. Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94$\times$ throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.

Problem

• Mixture-of-Experts
• LLM serving
• GPU memory
• KV cache
• weight quantization

Innovation

• PagedWeight
• Mixture-of-Experts
• dynamic quantization
• KV cache
• quality-aware