LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

2026-08-04

AI Summary

This study systematically investigates the scaling laws of Mixture-of-Experts (MoE) diffusion language models, addressing critical theoretical gaps in optimization, data-to-model allocation, and architectural design. It reveals, for the first time, quantitative differences in scaling behavior between MoE diffusion models and autoregressive counterparts, and introduces IsoFLOP—a computation allocation strategy tailored for diffusion language models—alongside principles for stable expert architecture design. Leveraging these insights, the authors train LLaDA MoE v2 from scratch, featuring 30 billion total parameters with only 3 billion activated per token, using merely 65% of Qwen3’s pretraining data (23.5T tokens). The model matches Qwen3’s performance across diverse knowledge, reasoning, and code tasks; after supervised fine-tuning, it surpasses SDAR Chat on seven out of eight benchmarks.

Abstract

Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65\% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.

Problem

• diffusion language models
• Mixture-of-Experts
• scaling laws
• compute allocation
• model architecture

Innovation

• Mixture-of-Experts
• diffusion language models
• scaling laws
• IsoFLOP analysis
• expert architecture