Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation

2026-09-17

AI Summary

研究针对自回归图像生成中KV缓存访问瓶颈问题，通过分析注意稀疏性特征，提出对角线感知稀疏注意机制，显著提高处理速度和效率。

Abstract

Autoregressive image generation has emerged as a paradigm for multimodal AI systems due to its compatibility with transformer-based LLM serving infrastructures. However, generating thousands of visual tokens per request makes decoding increasingly bottlenecked by KV cache accesses during attention computation. Sparse attention is particularly attractive for this workload because many visual generation applications tolerate moderate quality degradation in exchange for improved performance and efficiency. While sparse attention has been extensively explored for text-based LLM inference, it remains unclear whether its sparsity assumptions generalize effectively to autoregressive image generation. We present the first systematic characterization of attention sparsity in autoregressive image generation across diverse workloads and representative open-source models. Our analysis reveals several distinguishing properties, including a pronounced prefill-decode asymmetry, strong attention concentration on prompt and local tokens, and a unique diagonal attention sparsity pattern arising from the spatial locality of visual tokens. Motivated by these observations, we propose a diagonal-aware sparse attention mechanism that selectively skips KV entries along the diagonal attention direction within a recent window. Implemented on top of a GPU-based serving system using FlexGen, FlashAttention-2, and custom kernels, our approach achieves up to 3.1x throughput and 1.19x latency improvements with less than 2% quality degradation compared to dense inference.

Problem

• Autoregressive Image Generation
• Attention Sparsity
• KV Cache Accesses

Innovation

• diagonal-aware sparse attention
• autoregressive image generation
• KV cache access optimization
• attention sparsity