ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation

2026-09-23

AI Summary

This study addresses the excessive resource consumption caused by reliance on teacher and critic networks in few-step autoregressive video generation. To this end, it proposes a teacher-free and critic-free video post-training scheme. Methodologically, this work pioneers a generator-only post-training paradigm that achieves efficient, low-latency streaming generation via representation distribution matching. It introduces dynamic regularization to enforce temporal consistency and overcomes memory bottlenecks by integrating stochastic truncation supervision, a lightweight VAE decoder, and staged vector-Jacobian product techniques. Experimental results demonstrate that the proposed method requires only 16 GPU hours to substantially enhance video quality, achieving a VBench score of 84.87 and outperforming existing baselines.

Abstract

Few-step autoregressive (AR) video diffusion enables low-latency streaming generation, but existing post-training methods predominantly rely on Distribution Matching Distillation (DMD), requiring both a large pretrained teacher and an online critic to estimate distributional discrepancies through diffusion scores. In this work, we ask whether this resource-intensive teacher--critic stack can be eliminated by post-training only the generator against a precomputed target distribution. Drawing inspiration from representation distribution matching (RDM) for one-step image generation, we systematically study its transfer to few-step causal video generation and identify three key barriers: a memory-intractable gradient path, a distinct video optimization regime, and representation distributions that underconstrain temporal dynamics. We introduce ViRDM, a teacher- and critic-free video post-training recipe that addresses these barriers sequentially. By coupling RDM with stochastically truncated clean-exit supervision, a lightweight VAE decoder, and staged vector--Jacobian products, ViRDM makes representation distribution matching memory-feasible for multi-step causal video rollouts. We further establish effective generated-population and initialization regimes for video RDM, and introduce lightweight dynamics regularization to compensate for the underconstrained temporal dynamics. ViRDM turns three-network distillation into generator-only post-training, reducing GPU memory use and training time while improving video quality. With only 20 generator updates, the recipe reaches 84.87 on the official VBench evaluation, outperforming the previous best few-step causal baseline by 0.36, while requiring 16 A100 GPU-hours. We additionally report exploratory results demonstrating the potential of the same recipe for lower causal sampling budget and for one-, two-, and four-step bidirectional generation.

Problem

• Few-step video generation
• Distribution Matching Distillation
• Representation Distribution Matching
• Autoregressive video diffusion
• Post-training

Innovation

• Representation Distribution Matching
• Few-Step Causal Video Generation
• Teacher-Free Post-Training
• Memory-Efficient Optimization
• Dynamics Regularization