FastVR: Efficient Streaming Video Restoration with One-Step Diffusion

2026-09-29

AI Summary

This study addresses the deployment bottlenecks in diffusion-based video inpainting caused by the substantial overhead of VAE encoding-decoding and the quadratic scaling of DiT self-attention computation. To overcome these limitations, this work proposes a one-step diffusion-based streaming video inpainting framework. Methodologically, it introduces a lightweight VAE alongside a chunked causal attention mechanism to significantly reduce inference complexity, while incorporating velocity consistency regularization and continuous trajectory learning to enhance generation quality and temporal coherence. Experimental results demonstrate that the proposed approach achieves state-of-the-art performance on both synthetic and real-world benchmarks. Furthermore, it enables real-time processing of 1080p videos at 11 FPS on a single H20 GPU, comprehensively outperforming existing baselines in computational efficiency.

Abstract

Diffusion-based video restoration recovers realistic details, but its practical deployment is limited by two efficiency bottlenecks: costly VAE encoding and decoding, and the quadratic cost of full self-attention in diffusion transformers (DiTs). This paper presents FastVR, a streaming video restoration framework built on a one-step diffusion model, which delivers strong restoration quality and temporal consistency while processing 1080p video at 11 FPS on a single H20 GPU. To improve inference efficiency, FastVR combines a lightweight VAE with chunk-wise causal attention, which substantially reduces the computational cost. During training, it further adopts velocity consistency regularization and continuous trajectory learning, which improve restoration quality. Extensive experiments show that FastVR is more efficient than the evaluated diffusion baselines while achieving state-of-the-art performance on synthetic and real-world benchmarks. We hope that this work supports further progress in the community.

Problem

• Video Restoration
• Diffusion Models
• Computational Efficiency
• VAE Bottleneck
• Self-Attention Cost

Innovation

• One-Step Diffusion
• Streaming Video Restoration
• Chunk-wise Causal Attention
• Lightweight VAE
• Velocity Consistency Regularization