GPU Kernel Scientist: An LLM-Driven Framework for Iterative Kernel Optimization

2025-06-25

AI Summary

Addressing the challenge of kernel optimization on next-generation GPUs (e.g., AMD MI300), where scarce documentation and limited expert knowledge hinder manual tuning, this paper proposes an LLM-driven, multi-stage evolutionary agent framework. The framework operates without human priors, autonomously generating optimization hypotheses, evolving CUDA/HIP code variants, scheduling experiments, and iterating closed-loop based solely on runtime performance feedback. It integrates general GPU optimization knowledge from literature, architecture-specific adaptation mechanisms, and external evaluation systems to enable end-to-end automated tuning. Experiments demonstrate its feasibility in complex heterogeneous hardware environments, substantially reducing reliance on domain experts. To our knowledge, this is the first work to deeply apply LLMs to low-level, system-level performance optimization—establishing a novel paradigm for efficient programming of emerging accelerator architectures.

Technology Category

Search and Optimization: Learning to SearchMachine Learning: Hardware-aware MLCognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Optimizing GPU kernels for high performance is a complex task, often demanding deep architectural knowledge, extensive profiling, and iterative experimentation. This challenge is amplified when targeting newer or less-documented GPU architectures where traditional development aids are scarce. This paper introduces an LLM-powered "GPU Kernel Scientist," an automated methodology for iteratively refining accelerator kernels. Our methodology employs LLMs in a multi-stage, evolutionary process: (a) strategically selecting promising prior code versions as a basis for new iterations; (b) generating hypotheses for optimization experiments, based on existing code and assimilated knowledge from general GPU literature; and (c) autonomously implementing these experiments through code modification and subsequent submission to an external evaluation system, using only observed timing data as performance feedback. We detail how this approach navigates the challenges of the AMD MI300 target architecture and leverages LLMs to compensate for limited domain-specific human expertise. Since quantitative results from an ongoing performance competition were embargoed on paper submission date, we present the architectural design, operational workflow, and qualitative insights, highlighting the potential of LLM-driven agents to democratise and accelerate GPU kernel optimization, especially in resource-constrained or rapidly evolving hardware environments.

Problem

• Automating GPU kernel optimization for high performance
• Addressing challenges in newer or less-documented GPU architectures
• Leveraging LLMs to reduce need for domain expertise

Innovation

• LLM-driven iterative GPU kernel optimization
• Multi-stage evolutionary code refinement
• Autonomous hypothesis testing via timing feedback