Token Radius Attention for Efficient Video Generation

2026-08-03

AI Summary

This work addresses the high computational cost of 3D self-attention in video diffusion Transformers and the limitation of existing sparse attention methods, which ignore the varying attention resource demands across queries. The authors propose Token Radius Attention, a training-free framework that, for the first time, reveals a log-linear relationship between query retention density and attention entropy. Leveraging this insight, they design an entropy-driven adaptive sparsification mechanism that dynamically allocates attention budgets based on query entropy, translating them into time-decaying local neighborhood radii without explicit key sorting. Integrated with entropy extraction, warm-up reuse, and block-sparse masking, the method retains only 9–19% of attention interactions across seven models—including Wan2.1/2.2 and HunyuanVideo—achieving 1.56–2.05× speedup while preserving generation quality.

Abstract

Video Diffusion Transformers (VDiTs) enable high-fidelity generation but incur quadratic cost from dense 3D self-attention. Existing head- and block-level sparse methods share computation budgets across queries, overlooking token-specific attention demand. We observe that retained density varies across queries yet correlates log-linearly with attention entropy, while dominant interactions form query-centered neighborhoods with token-dependent radii. Based on these findings, we propose Token Radius Attention (TRA), a training-free framework that maps query entropy to an analytic token budget and converts it into a temporally decayed radius without explicit key ranking. Fused entropy extraction, warm-up reuse, and block-sparse mask construction further reduce overhead. Across seven Wan2.1, Wan2.2, and HunyuanVideo T2V/I2V configurations, TRA retains only 9-19% of attention interactions and achieves 1.56x-2.05x speedup with competitive generation quality. Code is available at https://github.com/IF-LAB-PKU/Token-Radius-Attention.

Problem

• Video Diffusion Transformers
• self-attention
• computational efficiency
• token-specific attention
• quadratic complexity

Innovation

• Token Radius Attention
• Video Diffusion Transformers
• Sparse Attention
• Attention Entropy
• Training-Free Acceleration