Decoupling Logical Masks from GPU Execution for Dynamic Block-Sparse Attention

2026-09-22

AI Summary

本文提出Tessera，通过解耦逻辑掩码与GPU执行来优化动态块稀疏注意力机制，以提高视频扩散模型中注意力计算的效率。

Abstract

Attention computation makes inference expensive in video diffusion transformers (vDiTs), which generate videos through iterative denoising. Block-sparse attention (BSA) reduces this cost by computing only blocks selected by a logical mask, which specifies attention interactions to compute. However, coupling logical block geometry to execution choices limits adaptation to varying masks and graphics processing units (GPUs), while runtime kernel specialization can incur preparation overhead that outweighs execution time savings. We present Tessera, a specialized runtime for dynamic BSA that decouples logical masks from GPU execution while preserving specified attention interactions. Its physical mapping layer retains, combines, or subdivides logical attention blocks into physical tiles suited to different attention mask shapes and GPU architectures. Its task organization layer groups and schedules tiles within GPU tasks to reuse data, expose parallelism, and overlap data movement with computation. Finally, profile-guided regime selection enables low- overhead execution plan selection through a lookup table constructed from offline profiling. We implement Tessera with specialized CUDA kernels supporting four NVIDIA GPU generations. Evaluated on 2,315 real attention masks and industrial video diffusion models, Tessera achieves up to 6.79x BSA request speedup over baseline systems in the evaluated video diffusion models.

Problem

• video diffusion transformers
• block-sparse attention
• logical mask
• GPU execution
• runtime kernel specialization

Innovation

• Dynamic Block-Sparse Attention
• Decoupling Logical Masks
• Physical Mapping Layer
• Task Organization Layer
• Profile-Guided Regime Selection