SparseDitto: Customizing GPU Kernels for Different Sparsity Patterns with LLM-Based Agentic System

2026-08-05

AI Summary

This work addresses the challenge of achieving high performance for sparse matrix computations across diverse sparsity patterns, operators, and GPU architectures. To this end, it presents the first end-to-end adaptive system that integrates a large language model (LLM) agent with a lightweight structural feature model to automatically generate high-performance kernels tailored to specific sparsity patterns, operators, and target GPUs. The system unifies support for SpMV, SpMM, and SpGEMM through a structural-feature-driven strategy ranking module, an architecture-aware planner, and a code-generation-and-verification agent. Experimental results demonstrate average speedups of 2.68× and 2.79× on RTX PRO 6000 and H200 GPUs, respectively, with peak acceleration reaching 146.61×, and up to 3.39× speedup in full-batch GCN training.

Abstract

Sparse matrix kernels are fundamental to scientific computing, graph analytics, and machine learning. Their GPU performance depends strongly on the input sparsity pattern and execution strategy. For the same SpMM on the same matrix, cuSPARSE exhibits a 350x performance gap between CSR and Blocked-ELL. Our study of multiple data formats, specialized systems, and sparse compilers shows that no single implementation consistently dominates across sparsity patterns and operators. This motivates a system that can adapt its representation, execution strategy, and hardware mapping to each workload and target GPU. We present SparseDitto, an LLM-based system that constructs a GPU kernel for each matrix, operator, and target GPU. SparseDitto supports SpMV, SpMM, and SpGEMM within a unified design framework. A lightweight additive model ranks established strategies using structural features of the input matrix. An architecture-aware planner then proposes several candidate designs. Coding and verification agents implement and refine them using measurements from the target GPU. Across three sparse operators and a diverse set of matrices, SparseDitto achieves a geometric-mean speedup of 2.68x over cuSPARSE on an NVIDIA RTX PRO 6000 GPU, with a maximum of 146.61x. On an NVIDIA H200 GPU, it achieves 2.79x, with a maximum of 78.5x. Its generated SpMM kernels also accelerate full-batch GCN training by up to 3.39x.

Problem

• sparse matrix
• GPU kernel
• sparsity pattern
• performance optimization
• automatic customization

Innovation

• LLM-based agentic system
• adaptive sparse kernels
• sparsity pattern customization
• GPU kernel generation
• hardware-aware optimization