DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation

2026-06-03

AI Summary

This work addresses the inefficiency of autoregressive video diffusion models, whose fixed denoising schedules often lead to either computational redundancy or insufficient refinement, hindering real-time generation. To overcome this limitation, the authors propose DSA, a confidence-guided dynamic computation framework that introduces adaptive step allocation into such models for the first time. DSA employs a lightweight confidence head to predict the reliability of denoising at each frame and jointly trains the generator and confidence head using a distribution-matching distillation objective. During inference, the model adaptively terminates or continues denoising based on predicted confidence, without requiring additional data or heuristic rules. Evaluated on an H100 GPU, the method achieves 22.63 FPS with sub-second latency while matching or surpassing state-of-the-art models in VBench quality metrics.

Abstract

Video diffusion transformers have achieved state-of-the-art visual quality, but their high inference cost remains a major bottleneck for real-time applications. Recent distillation frameworks produce autoregressive video diffusion models with reduced latency, yet these models still use a fixed number of denoising steps per frame, wasting computation on predictable frames and under-refining challenging ones. We present DSA, a confidence-guided adaptive computation framework for AR video diffusion. DSA introduces a lightweight confidence head, trained jointly with the generator under a distribution-matching distillation objective, to estimate per-frame denoising reliability. At inference, this confidence signal dynamically adjusts the number of diffusion steps: simple frames terminate early for speed, while complex frames receive additional refinement. Our method requires no extra video data, no heuristics, and little architectural modification. Experiments show that DSA achieves real-time autoregressive video generation, reaching 22.63 FPS with sub-second latency on H100 GPUs, while maintaining competitive or superior VBench quality compared to recent autoregressive and bidirectional video diffusion models. Our results demonstrate that confidence-guided adaptive sampling provides an effective and practical path toward interactive video generation.

Problem

• autoregressive video generation
• diffusion models
• adaptive computation
• inference efficiency
• real-time video generation

Innovation

• Dynamic Step Allocation
• Confidence-guided Inference
• Autoregressive Video Diffusion
• Adaptive Computation
• Video Generation