GRACE: Generation-aware latent compression for efficient video generation

2026-10-07

AI Summary

This study addresses the reconstruction degradation and distribution shift issues caused by highly compressed video autoencoders when integrated with pretrained Diffusion Transformers (DiTs), which hinder efficient video generation. We introduce a novel generation-aware latent compression mechanism within a two-stage adaptation framework. Specifically, we freeze base latents while learning residual latents aligned in the DiT feature space to eliminate distribution shift, followed by lightweight fine-tuning combined with an asymmetric denoising strategy for efficient transfer. Crucially, our approach avoids training from scratch. Applied to Wan2.1-I2V-14B, it achieves an 8× token reduction and an 11.1× latency decrease while preserving VBench generation quality comparable to the original model.

Abstract

Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far fewer tokens. However, such autoencoders are challenging to train, since a higher compression ratio degrades reconstruction quality and recovering it requires more channels, which is known to slow the convergence of the DiT. The compressed latent also differs from the one the DiT was trained on, so the pretrained DiT must be either retrained from scratch or adapted at considerable cost. Compressing the autoencoder the DiT was trained with appears to preserve compatibility, yet optimizing it for reconstruction alone still shifts the latent away from the distribution the DiT has learned. To address this, we propose Generation-Aware Latent Compression for Efficient Video Generation (GRACE), a two-stage framework that compresses a pretrained video autoencoder while keeping it compatible with the pretrained DiT. Specifically, we keep a frozen base latent from the pretrained encoder and learn a residual latent for the information lost under stronger compression, while aligning the compressed latent with the pretrained latent in the feature space of the frozen DiT so that the autoencoder is optimized for generation. We then adapt the DiT with lightweight fine-tuning and asymmetric denoising, where the base is denoised ahead of the residual. GRACE reduces the token count of Wan2.1-I2V-14B by 8x and its latency by 11.1x at 480x832x81, while matching the generation quality of the pretrained pipeline before compression on VBench.

Problem

• Video Generation
• Latent Compression
• Video Autoencoder
• Diffusion Transformer
• Generation Compatibility

Innovation

• Latent Compression
• Video Autoencoder
• Diffusion Transformer
• Residual Learning
• Asymmetric Denoising