Opal: A Modular Framework for Optimizing Performance using Analytics and LLMs

2025-10-01

AI Summary

Large language models (LLMs) struggle to autonomously optimize GPU kernel code without performance context, while conventional profiling tools identify bottlenecks but fail to generate executable optimization strategies. Method: This paper introduces the first GPU kernel auto-optimization framework integrating LLMs with dynamic hardware performance insights—including hardware counter metrics and Roofline model analysis—via structured prompting that precisely encodes bottleneck characteristics to guide high-fidelity, executable code generation. Contribution/Results: Evaluated across 1,640 experiments, the framework achieves performance improvements in over 98.5% of cases, with average speedups ranging from 19.34% to 52.3%; generated code exhibits near-perfect functional correctness. This work bridges the longstanding gap between low-level performance analysis and high-level optimization decision-making, establishing a reproducible, scalable paradigm for AI-driven, system-level GPU code optimization.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Hardware-aware ML
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) show promise for automated code optimization but struggle without performance context. This work introduces Opal, a modular framework that connects performance analytics insights with the vast body of published by guiding LLMs to generate informed, trustworthy optimizations. Unlike traditional performance tools that identify bottlenecks but stop short of actionable suggestions, Opal bridges this long-standing gap by linking dynamic insights from hardware counters and Roofline analysis to stall events to optimization decisions. We evaluate Opal across 1640 experiments on real-world GPU kernels and find that in over 98.5% of cases, even a single insight source yields speedups, ranging on average from 19.34% to 52.3%. Our prompt template produced correct code in all but one case, where a vague diagnostic caused an unsafe suggestion. By automatically optimizing GPU kernels using performance analytics and LLMs, Opal marks a leap toward democratizing expert-level performance engineering for all.

Problem

• Automating code optimization using LLMs with performance context
• Bridging performance analysis insights to actionable optimization decisions
• Generating trustworthy GPU kernel optimizations via analytics-guided LLMs

Innovation

• Modular framework connects analytics with LLM optimizations
• Links hardware counters and Roofline analysis to decisions
• Automatically optimizes GPU kernels using performance analytics