Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generation

2026-07-23

AI Summary

This work addresses the limitation of existing reinforcement learning approaches for CUDA code generation, which rely solely on outcome-based signals such as correctness and speedup while neglecting program structural properties critical to performance. To bridge this gap, the authors propose CudaPerf, a novel framework that incorporates structure-aware rewards—such as memory coalescing, occupancy, and arithmetic intensity—into the reinforcement learning process, alongside verifiable execution-based rewards. By jointly optimizing correctness, performance, and structural efficiency through offline comparative ranking and online multi-round reinforcement learning, and iteratively refining outputs using execution feedback, CudaPerf achieves substantial improvements. Experiments demonstrate up to 5× and 3.32× speedup gains over strong baselines like Qwen-3-32B and CUDA Agent on C-to-CUDA and PyTorch-to-CUDA tasks, respectively, along with 17% and 7% higher correctness rates.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful technique to enhance the reasoning capacity of LLMs for optimized code generation. However, existing RLVR approaches primarily rely on outcome-based signals such as correctness and speedup, overlooking performance-critical structural properties of programs that are essential for generating optimized code. In this work, we propose CudaPerf, a reflective RL framework that incorporates both verifiable execution rewards and structural code-aware rewards derived from parallelization features (e.g., memory coalescing, occupancy, Arithmatic Intensity, and synchronization patterns). CudaPerf operates in two stages: (1) an offline pairwise ranking module that learns to distinguish strong and weak program candidates via contrastive comparisons, and (2) an online RL training phase that jointly optimizes for correctness, performance, and structural efficiency through a unified reward signal. To further enhance learning, CudaPerf utilizes iterative refinement using execution feedback enabling progressive improvement of generated candidates. We also introduce a dataset comprising 2.9k C to CUDA and 1k PyTorch to CUDA programs, each paired with diverse input configurations and multiple CUDA implementations encompassing diverse optimization strategies. CudaPerf is evaluated across multiple benchmarks comprising both C to CUDA and PyTorch to CUDA transformations. Empirical findings suggest that CudaPerf significantly outperforms strong baselines, including Qwen-3-32B (for C to CUDA) and CUDA Agent (for PyTorch to CUDA) by achieving up to 5X & 3.32X improvements in speedup, and 17% & 7% improvements in correctness, respectively.

Problem

• CUDA kernel generation
• reinforcement learning
• structural code properties
• performance optimization
• verifiable rewards

Innovation

• Reinforcement Learning with Verifiable Rewards
• Structural Code-aware Rewards
• CUDA Kernel Generation
• Multi-turn RL
• Performance-Aware Code Optimization