Regular-Activation Concentration: Characterizing Column-Level Output Sparsity Across Diffusion Model Architectures

2026-05-30

AI Summary

This work addresses a critical mismatch between existing diffusion model acceleration techniques—which rely on element-wise activation sparsity—and the column-granularity data processing inherent in modern hardware, leading to overestimated practical sparsity benefits. The study presents the first systematic characterization of output sparsity across seven diffusion models at the column level, identifying three distinct activation distribution patterns and uncovering the interplay between model architecture and memory layout optimization. Through hardware-aware column-level sparsity profiling, cycle-accurate GDDR6 simulation, multi-threshold accuracy evaluation, and cross-modal comparison, the authors demonstrate that memory stalls account for 84–89% of total execution cycles and that element-wise sparsity poorly predicts actual hardware gains. Their approach achieves up to 30.6% reduction in execution cycles on UNet+Transformer models, with a maximum latency decrease (MLD) of 50.8%.

Abstract

Recent diffusion accelerators exploit activation sparsity by skipping near-zero GELU outputs, reporting 52--85% element-level sparsity. However, systolic-array hardware processes activations at column granularity, where a single non-zero element forces the entire column to be computed. We present the first systematic column-level sparsity characterization across seven diffusion workloads spanning three workload groups and four modalities. Our measurements reveal that element-level sparsity overstates hardware-exploitable sparsity by up to 78 percentage points and exposes a three-way taxonomy. UNet+transformer workloads exhibit activation concentration with workload-dependent cycle reductions up to 30.6%. Pure-transformer DiT shows dispersion, yielding 12.4%. Motion/dance transformer workloads range from modest reductions to 50.8% for MLD, driven by its extreme token dimension and expansion ratio. Cycle-level simulation on a GDDR6-based accelerator confirms that memory stalls account for up to 84--89% of total cycles and that layout sensitivity tracks the profiling-based taxonomy. A full accuracy sweep across five thresholds reveals that UNet+transformer workloads degrade gracefully, while motion models exhibit an accuracy cliff between the primary operating point and the next threshold. Our characterization shows that workload group and model dimensions jointly determine whether column-level memory layout optimization is beneficial, and element-level sparsity alone is insufficient for that prediction.

Problem

• activation sparsity
• column-level sparsity
• diffusion models
• systolic array
• memory layout

Innovation

• column-level sparsity
• diffusion models
• activation concentration
• memory layout optimization
• systolic array