MMOE: Modernizing Diffusion Transformers with Efficient Expert Design

2026-07-27

AI Summary

This work addresses the common oversight in existing AIGC diffusion models that prioritize parameter count and sparsity at the expense of balanced training and deployment costs when incorporating sparse mixture-of-experts mechanisms. It presents the first systematic adaptation and optimization of efficient sparse expert architectures—well-established in large language models—to diffusion Transformers, introducing the MMOE architecture. MMOE integrates routed experts, shared lightweight experts, gated residual routing, and attention residual reuse. Evaluated under a single-node, 8×H100 GPU training budget, MMOE achieves significantly faster convergence and consistently outperforms baseline models across all checkpoints in terms of FID, establishing a new state-of-the-art trade-off between generation quality and computational efficiency among sparse diffusion models. Routing analysis further reveals stable expert specialization and effective activation of lightweight pathways.

Abstract

Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.

Problem

• AIGC Foundation Models
• diffusion transformers
• sparse experts
• efficiency
• scaling

Innovation

• Mixture of Experts
• Diffusion Transformer
• Efficient Scaling
• Lightweight Experts
• Gate-Residual Routing