DiffusionGemma Technical Report

2026-07-31

AI Summary

This work addresses the inference speed bottleneck inherent in autoregressive large language models caused by sequential token-by-token generation. It introduces, for the first time, a discrete diffusion mechanism into large-scale language modeling, proposing a two-stage efficient training pipeline based on the Gemma-4 Mixture-of-Experts architecture. The approach integrates supervised fine-tuning, reinforcement learning, and sampler distillation to enable bidirectional denoising and parallel block-wise token generation. With a single forward pass, the model generates approximately 20 tokens, achieving a throughput of 1500 tokens per second on a single H100 GPU—significantly outperforming existing autoregressive baselines. Crucially, it preserves strong capabilities in long-context understanding, multimodal reasoning, and complex inference, while also supporting hybrid diffusion–autoregressive decoding strategies.

Abstract

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

Problem

• autoregressive models
• sequential decoding bottleneck
• text generation speed
• diffusion-based generation
• language model efficiency

Innovation

• discrete diffusion
• parallel token generation
• two-stage fine-tuning
• sampler distillation
• hybrid decoding