🤖 AI Summary
Existing video generation methods based on latent diffusion models suffer from high computational costs, hindering real-time applications. This work introduces, for the first time, the concept of inter-frame redundancy reduction—long employed in conventional video compression—into diffusion Transformer architectures. The authors propose an inter-frame latent pruning strategy that requires no additional training and design an attention restoration mechanism to mitigate visual artifacts introduced by pruning. This approach effectively bridges the inconsistency between training and inference, achieving a video editing throughput of 12.44 FPS on an NVIDIA RTX 6000 GPU—1.44× faster than the baseline—while preserving generation quality.
📝 Abstract
Video generation, while capable of generating realistic videos, is computationally expensive and slow, prohibiting real-time applications. In this paper, we observe that video latents encoded via an autoencoder under the Latent Diffusion Model (LDM) framework contain redundancy along the temporal axis. Analogous to how traditional video compression algorithms avoid transmitting redundant frame data, we propose the Latent Inter-frame Pruning framework to prune (skip the re-computation of) duplicated latent patches, thereby reducing computational burden and increasing throughput. However, direct pruning results in visual artifacts due to the discrepancy between full-sequence training and pruned inference. To resolve these artifacts, we propose an Attention Recovery mechanism to bridge the train-inference gap. With our proposed method, we increase video editing throughput by 1.44$\times$, achieving 12.44 FPS on an NVIDIA RTX 6000 while maintaining video quality. We hope our work inspires further research into integrating traditional video compression methods with modern video generation pipelines. This work is a preliminary work on Training-free Latent Inter-Frame Pruning with Attention Recovery.