MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance

2025-05-02

AI Summary

Post-training quantization of Mixture-of-Experts (MoE) large language models suffers from dual distributional imbalance—both across experts (inter-expert) and within individual experts (intra-expert)—leading to insufficient calibration, increased quantization error, and degraded generalization. To address this, we propose the Expert-Balanced Self-Sampling and Affinity-Guided Quantization (EBSS-AGQ) framework. First, we introduce Expert-Balanced Self-Sampling (EBSS), a novel calibration data selection strategy that achieves cross-expert balance by jointly optimizing token accumulation probability and expert load fairness. Second, we design Affinity-Guided Quantization (AGQ), which explicitly models the dynamic expert–sample affinity during routing and incorporates layer-aware error compensation. Evaluated on DeepSeekMoE-16B, our 4-bit quantized model achieves over a 10-point improvement in HumanEval accuracy, substantially outperforming existing post-training quantization (PTQ) methods. To our knowledge, this is the first work to systematically resolve quantization imbalance under MoE’s sparse routing paradigm.

Technology Category

Machine Learning: Mixture of Experts (MoE)Search and Optimization: Sampling/Simulation-based SearchNatural Language Processing: (Large) Language Models

Application Category

User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systemsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingEconomics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Mixture-of-Experts (MoE) large language models (LLMs), which leverage dynamic routing and sparse activation to enhance efficiency and scalability, have achieved higher performance while reducing computational costs. However, these models face significant memory overheads, limiting their practical deployment and broader adoption. Post-training quantization (PTQ), a widely used method for compressing LLMs, encounters severe accuracy degradation and diminished generalization performance when applied to MoE models. This paper investigates the impact of MoE's sparse and dynamic characteristics on quantization and identifies two primary challenges: (1) Inter-expert imbalance, referring to the uneven distribution of samples across experts, which leads to insufficient and biased calibration for less frequently utilized experts; (2) Intra-expert imbalance, arising from MoE's unique aggregation mechanism, which leads to varying degrees of correlation between different samples and their assigned experts. To address these challenges, we propose MoEQuant, a novel quantization framework tailored for MoE LLMs. MoE-Quant includes two novel techniques: 1) Expert-Balanced Self-Sampling (EBSS) is an efficient sampling method that efficiently constructs a calibration set with balanced expert distributions by leveraging the cumulative probabilities of tokens and expert balance metrics as guiding factors. 2) Affinity-Guided Quantization (AGQ), which incorporates affinities between experts and samples into the quantization process, thereby accurately assessing the impact of individual samples on different experts within the MoE layer. Experiments demonstrate that MoEQuant achieves substantial performance gains (more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization) and boosts efficiency.

Problem

• Addresses memory overhead in MoE LLMs via quantization.
• Solves inter-expert imbalance in MoE model calibration.
• Mitigates intra-expert imbalance from MoE aggregation mechanisms.

Innovation

• Expert-Balanced Self-Sampling for balanced calibration
• Affinity-Guided Quantization incorporating expert-sample affinities
• Novel quantization framework for MoE LLMs