KernelGenBench: A Multi-Source and Multi-Chip Benchmark for LLM-based Kernel Generation

2026-07-22

AI Summary

Current large language models (LLMs) lack a unified evaluation benchmark for generating efficient and portable accelerator kernels. This work proposes KernelGenBench, the first comprehensive benchmark for evaluating Triton kernel generation across diverse operators and hardware platforms, comprising two sub-benchmarks: MS (multi-source) and MC (multi-chip). Leveraging a large-scale operator dataset, LLM- and agent-based generation frameworks, and heterogeneous hardware performance evaluation, the benchmark systematically assesses kernel generation methods in terms of performance, portability, and cost efficiency. Experimental results demonstrate that agent-based approaches outperform pure LLM sampling; however, performance degrades significantly across platforms. On average, generating a single valid kernel consumes 5.11 million tokens, highlighting critical bottlenecks in generalization capability and computational cost of current methods.

Abstract

Large language models (LLMs) have significantly increased the demand for efficient accelerator kernels, but kernel development remains a highly specialized and labor-intensive task. The recent rise of LLMs and agentic frameworks offers a promising pathway toward automatic kernel generation. However, despite rapid progress, there is still no comprehensive benchmark to rigorously evaluate LLM-generated kernels across diverse operator sources or heterogeneous hardware platforms. We present KernelGenBench, a unified benchmark for systematically evaluating LLM- and agent-generated Triton kernels across diverse operator sources and heterogeneous hardware platforms. It comprises two complementary sub-benchmarks: KernelGenBench-MS (Multi-Source), evaluating 210 operators from three sources beyond standard PyTorch-centric tasks, and KernelGenBench-MC (Multi-Chip), measuring performance portability across six heterogeneous hardware platforms using a 110-operator subset. Our large-scale evaluation, consuming over 15 billion tokens, shows: (1) agent-based methods consistently outperform pure LLM sampling methods, while cuBLAS operators are the most challenging across all methods; (2) generation performance varies significantly across hardware platforms, with even recent kernel-specialized agents experiencing severe cross-platform degradation (e.g., AutoKernel drops from 87% on NVIDIA to 25% on Platform E); (3) autonomous kernel generation remains highly cost-intensive, with specialized agent methods averaging 5.11 million tokens per successful operator (AKO4all reaches 5.19 million), orders of magnitude higher than simple LLM sampling approaches.

Problem

• LLM-based kernel generation
• benchmark
• heterogeneous hardware
• multi-source operators
• performance evaluation

Innovation

• kernel generation
• large language models
• benchmark
• heterogeneous hardware
• multi-source operators