TIMERIPPLE: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

2025-11-15

AI Summary

Current video diffusion Transformers (vDiTs) suffer from high inference latency due to redundant self-attention computations. Moreover, prevailing sparsification methods—borrowed naively from language models—overlook the intrinsic spatiotemporal correlations inherent in video data. To address this, we propose a lightweight, adaptive channel-wise attention score reuse strategy: by modeling token-level spatiotemporal dependencies in the latent space, our method selectively reuses partial attention scores from spatially or temporally adjacent tokens, thereby avoiding full attention computation. Crucially, it requires no predefined sparsity pattern, ensuring both generality and efficiency. Evaluated on four vDiT architectures, our approach achieves up to 85% FLOPs reduction while incurring less than 0.06 absolute degradation in VBench quality—a substantial improvement over existing acceleration techniques.

Technology Category

Computer Vision: Diffusion Models for VisionMachine Learning: Large Multimodal Models (LMMs)Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsSearch and Retrieval-Augmented AI: Efficiency and scalability of Web search enginesSystems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

The recent surge in video generation has shown the growing demand for high-quality video synthesis using large vision models. Existing video generation models are predominantly based on the video diffusion transformer (vDiT), however, they suffer from substantial inference delay due to self-attention. While prior studies have focused on reducing redundant computations in self-attention, they often overlook the inherent spatio-temporal correlations in video streams and directly leverage sparsity patterns from large language models to reduce attention computations. In this work, we take a principled approach to accelerate self-attention in vDiTs by leveraging the spatio-temporal correlations in the latent space. We show that the attention patterns within vDiT are primarily due to the dominant spatial and temporal correlations at the token channel level. Based on this insight, we propose a lightweight and adaptive reuse strategy that approximates attention computations by reusing partial attention scores of spatially or temporally correlated tokens along individual channels. We demonstrate that our method achieves significantly higher computational savings (85%) compared to state-of-the-art techniques over 4 vDiTs, while preserving almost identical video quality ($<$0.06% loss on VBench).

Problem

• Accelerating video diffusion transformers by reducing self-attention computational overhead
• Leveraging latent space spatio-temporal correlations to optimize attention mechanisms
• Developing adaptive reuse strategy for attention scores while preserving video quality

Innovation

• Leveraging spatio-temporal correlations in latent space
• Adaptive reuse strategy for attention computations
• Reusing partial scores of correlated tokens