At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

2026-07-28

AI Summary

This work addresses the lack of native support for the Gustavson dataflow’s index accumulation pattern in existing RISC-V Vector (RVV) architectures, which severely limits sparse tensor contraction performance far below the roofline limit. To overcome this, the authors propose Ventaglio—a runtime-configurable sparse execution unit—and extend the RVV instruction set to enable efficient gather-accumulate-scatter operations. This approach introduces, for the first time in RVV, a native index accumulation mechanism with only 3.1% area overhead, effectively alleviating software decoding and L1 fallback bottlenecks. Evaluated on a pruned LLaMA-3-8B model (DuoGPT), the design achieves speedups of 2.40–5.25× in prefill and 2.06–3.16× in autoregressive decoding stages, while core operators attain 6.9–7.4× acceleration over optimized baselines, closely approaching theoretical roofline performance.

Abstract

Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models. In the moderate-sparsity regime, Gustavson's dataflow provides a natural execution model for exploiting both activation and weight sparsity on vector processors through metadata-driven indexed accumulation. However, existing RVV architectures lack native support for this pattern, forcing kernels to rely on software index decoding and L1-backed indexed memory operations that keep sparse tensor contractions far below their roofline performance bound. We present Ventaglio, a runtime-configurable sparse execution unit coupled with RVV ISA extensions that drives sparse tensor contractions toward their roofline through indexed gather-accumulate-scatter support. Integrated into an open-source vector processing cluster and implemented in 12nm FinFET, Ventaglio accelerates sparse tensor contraction kernels by $6.9\text{--}7.4\times$ over optimized RVV baselines, with only $3.1\%$ area overhead for a cluster of tightly-L1 coupled vector processing elements. We build a performance-accurate instruction-level model of the Ventaglio extension, calibrate it against RTL implementation, and leverage it for scale-out performance analysis on a large $4\times4$ multi-cluster system. Using a DuoGPT-pruned LLaMA-3-8B model with practical $40\text{--}60\%$ dual sparsity, Ventaglio achieves $2.40\text{--}5.25\times$ and $2.06\text{--}3.16\times$ speedup over dense baselines during prefill and autoregressive decoding, respectively.

Problem

• sparse tensor contractions
• vector processors
• roofline performance
• RVV architecture
• Transformer inference

Innovation

• sparse tensor contraction
• vector processor
• RVV ISA extension
• roofline performance
• indexed gather-accumulate-scatter