Problem
Research questions and friction points this paper is trying to address.
Mixture-of-Experts
scratchpad-based tensor accelerators
expert weights
dataflow
A Multi-Engine Dataflow for MoE Decoding on Scratchpad-Based Tensor Accelerators
🤖 AI Summary
为解决MoE解码在基于暂存器的张量加速器上因专家权重移动导致的计算引擎闲置问题，提出CARDAN方法，通过向量量化和多引擎数据流设计提高效率。
📝 Abstract
Mixture-of-Experts (MoE) decoding on scratchpad-based tensor accelerators (STA) is dominated by moving expert weights while the compute engines sit idle. This traffic is hard to hide, because the experts are known only after routing, and hard to shrink without losing quality or adding critical-path work. We present CARDAN, which represents each expert-weight matrix as a vector-quantized component plus a shared-basis low-rank component and co-designs this representation with a multi-engine decoding dataflow. The representation separates expert-common from expert-private work, so the dataflow overlaps DMA with computation on several engines. Across five MoE families on AWS Trainium3, CARDAN matches or improves BF16-teacher perplexity across all five models and speeds up batch-one decoding by 1.15-1.31x over AWS dense MoE megakernels, rising to 1.7x at batch size 16.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Mixture-of-Experts
vector-quantized component
shared-basis low-rank component
multi-engine dataflow
scratchpad-based tensor accelerators
🔎 Similar Papers
No similar papers found.
