MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

2026-10-05

AI Summary

This study addresses the degradation in generation quality caused by attention mechanisms in diffusion Transformers under high sparsity. To overcome this limitation, this work proposes a training-free meta-cache sparse attention framework. The method introduces a novel meta-cache mechanism that reuses key-value selection and residual information, combined with precise probability-based selection and GPU-aligned grouped execution strategies. This approach eliminates the accuracy loss inherent in conventional sparse methods without requiring additional training. Experimental results demonstrate that the proposed framework achieves 1.8× and 2.32× speedups in video and 3D generation tasks, respectively, while preserving nearly lossless generation quality.

Abstract

Sparse attention is a primary approach to reducing the latency of diffusion transformers in long-sequence generation tasks, such as video and high-resolution 3D asset generation. However, existing methods can degrade generation quality and fidelity at high sparsity levels. Through controlled oracle comparisons, we trace this degradation to three sources: constraints imposed by token grouping, inaccurate interaction selection, and the attention contributions lost when tokens are discarded. Guided by this analysis, we propose Meta-Cached Sparse Attention (MC-Sparse), a training-free framework that selects individual key-value (KV) tokens while organizing similar queries into tile-aligned groups for efficient GPU execution. MC-Sparse caches metadata comprising query groups, KV indices selected using exact attention probabilities, and residuals between dense and sparse attention outputs, and reuses them across subsequent denoising steps. Across video and 3D generation models, MC-Sparse achieves higher fidelity to dense-attention outputs and larger denoising speedups than existing sparse-attention baselines, without visible quality degradation. Relative to dense attention, it delivers a $1.80\times$ denoising speedup on Minimax-H3-Base and a $2.32\times$ speedup on 3D asset generation, both with negligible quality loss.

Problem

• Sparse Attention
• Diffusion Transformers
• Long-sequence Generation
• Generation Fidelity
• Inference Latency

Innovation

• Sparse Attention
• Diffusion Transformers
• Training-free
• Metadata Caching
• KV Token Selection