FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformers

2026-07-16

AI Summary

This work addresses the high latency and memory overhead of existing latent video diffusion models that rely on 3D convolutional decoders, which hinder real-time inference for high-resolution or long-duration videos. The authors propose the first purely Transformer-based video decoder, enabling frame-by-frame streaming generation through a rolling key-value cache and temporal local attention mechanisms. This design ensures temporal causality without explicit causal masking and supports both training and inference at 1080p resolution. Evaluated in the Wan2.1/2.2 latent space, the method matches the reconstruction quality of convolutional baselines (achieving 41.55 dB PSNR versus 41.49 dB at 1080p), while accelerating decoding by 3.6–4.7× on a single H100 GPU and reducing memory consumption by up to 11×; with further inference optimizations, speedups reach 12×.

Abstract

Real-time video generation demands fast decoding as much as fast denoising, yet current latent video diffusion models rely on 3D convolutional decoders that are slow and memory-intensive at high resolutions or for long video. We introduce FlashDecoder, a fast, memory-efficient pure-Transformer video decoder that decodes latents to pixels frame by frame. At each step, the current frame attends only to a fixed-size window of past frames through a rolling KV cache. The fixed temporal window keeps decoding fast and memory bounded regardless of video length, enabling constant-latency streaming. Because frames are processed sequentially, temporal causality is enforced without explicit attention masks, enabling training at resolutions up to 1080p and matching the reconstruction quality of convolutional decoders. On the Wan2.1 and Wan2.2 latent spaces, FlashDecoder matches each convolutional decoder in reconstruction quality (e.g., 41.55dB vs. 41.49dB PSNR at 1080p) while decoding 3.6x-4.7x faster with up to 11x less memory on a single H100 GPU. With architecture-aware inference optimizations, the speedup widens to 12x.

Problem

• video decoding
• latent diffusion models
• real-time generation
• memory efficiency
• high-resolution video

Innovation

• FlashDecoder
• Transformer-based decoder
• rolling KV cache
• real-time video generation
• memory-efficient streaming