CommBench: Can LLMs Write Correct and Efficient GPU Communication Code?

2026-08-05

AI Summary

This work addresses the significant challenges large language models (LLMs) face in generating efficient and correct GPU communication code, which demands deep understanding of GPU architectures, network hardware, and distributed communication patterns. To this end, we introduce CommBench—the first benchmark comprising over 100 tasks spanning point-to-point, collective, and expert-parallel communication—and develop a cheat-resistant evaluation framework that supports automated compilation, execution, and verification. We present the first systematic assessment of LLMs on such tasks, proposing a unified metric that jointly considers functional correctness and communication performance, with validation conducted in real-world multi-GPU environments leveraging NVLink and RDMA. Experimental results reveal that even the strongest current model, GPT-5.5, satisfies both correctness and performance requirements on only 30.7% of tasks, highlighting a substantial gap between LLM capabilities and expert-level proficiency.

Abstract

Training and serving large language models (LLMs) rely heavily on high-performance GPU communication, yet implementing efficient GPU communication primitives requires deep expertise in GPU architectures, networking hardware, and distributed communication patterns, making them particularly challenging for code generation models. We present CommBench, a comprehensive benchmark for GPU communication programming, consisting of over 100 expert-curated tasks spanning point-to-point communication, collective operations, expert-parallel communication, compute--communication fusion, and communication utility functions, with reference implementations either written by GPU communication experts or distilled from production codebases. We further introduce a cheat-resistant evaluation framework that automatically compiles, executes, and validates generated code on multi-GPU systems, and a unified metric that jointly measures functional correctness and communication performance. Evaluating leading frontier and open-source code generation models on both intra-node NVLink and inter-node RDMA platforms reveals that even the strongest model, GPT-5.5, correctly implements and achieves competitive performance on only 30.7\% of the benchmark tasks. Our results expose a substantial gap between current LLMs and expert-written GPU communication code, establishing CommBench as a challenging benchmark for advancing AI-assisted systems programming.

Problem

• GPU communication
• code generation
• large language models
• distributed systems
• performance correctness

Innovation

• CommBench
• GPU communication
• code generation
• evaluation framework
• distributed systems