SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

2026-08-04

AI Summary

Video diffusion models suffer from inefficient inference under large-scale video tokens due to the high computational complexity of self-attention. This work proposes SPADE, a training-free sparse attention engine that introduces, for the first time, an input-adaptive dynamic sparsification mechanism. By dynamically selecting critical query, key, and value tokens, employing head-level sparsity strategies, and leveraging low-overhead execution, SPADE achieves substantial inference acceleration while preserving generation quality. Built upon the vDiT-SSR framework and integrating SICS runtime scheduling, Flash block-sparse attention, optimized index searching, and grouped kernel execution, SPADE delivers 2.26–3.40× speedup in attention computation and 1.49–1.80× end-to-end inference acceleration on Hunyuan-Video and Wan 2.1/2.2.

Abstract

Video diffusion transformers (vDiTs) generate high quality but pay quadratic self-attention cost, making inference prohibitive at video-token scales. The challenge is input-adaptive sparsity: selecting critical Q/K/V tokens with negligible overhead and executing them for end-to-end gains. We present SPADE, a training-free sparse-attention engine of three parts: (i) vDiT-SSR, a specification defining 3D blocking candidates and formalizing dynamic masks via Summarizer/Estimator expressions; (ii) runtime scheme generation using SICS and a head-wise policy; and (iii) an executor with low-overhead index search, flash block-sparse attention, and kernel grouping. Across Hunyuan-Video and Wan 2.1/2.2 for text-to-video and image-to-video generation, SPADE raises sparsity and speed while preserving quality, accelerating attention by 2.26x-3.40x and end-to-end inference by 1.49x-1.80x. Our code is open-sourced at https://github.com/6somehow/DAC-SPADE.

Problem

• video diffusion models
• self-attention
• input-adaptive sparsity
• inference efficiency
• quadratic complexity

Innovation

• sparse attention
• video diffusion transformers
• input-adaptive sparsity
• training-free acceleration
• block-sparse attention