dMoE: dLLMs with Learnable Block Experts

2026-05-29

AI Summary

This work addresses a critical mismatch in diffusion-based large language models (dLLMs) integrated with Mixture-of-Experts (MoE): block-wise parallel decoding conflicts with token-level expert selection, causing a surge in activated experts during inference and severe memory bottlenecks. To resolve this, the authors propose dMoE, the first framework that elevates expert routing from the token level to the block level. By aggregating token-level expert distributions within each block to produce a unified routing decision, dMoE enables coordinated expert activation across tokens. This approach reduces the average number of activated experts from 69.5 to 14.6—cutting memory consumption by 76.64%–79.84% and accelerating end-to-end latency by 1.14–1.66×—while preserving model performance at 99.11% of the original accuracy.

Abstract

Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide expert routing in a more coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% of the original performance. Meanwhile, it reduces memory usage by 76.64% to 79.84% and achieves 1.14$\times$ to 1.66$\times$ end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE

Problem

• Diffusion Large Language Models
• Mixture-of-Experts
• block parallel decoding
• expert routing
• memory bottleneck

Innovation

• dMoE
• Diffusion Large Language Models
• Mixture-of-Experts
• block-level routing
• parallel decoding