TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications

2026-06-03

AI Summary

Existing general-purpose code evaluation benchmarks struggle to assess the ability of large language models to generate and modify code within complex, state-sensitive, and mathematically rigorous telecommunications software stacks such as srsRAN 5G. To address this gap, this work introduces the first commit-driven benchmark tailored for telecom software engineering, extracting 734 structured test cases from the srsRAN 5G repository. The benchmark integrates executable unit tests with TeleJudge, a novel hierarchical LLM-based adjudication mechanism that evaluates context and semantic consistency at the file level, thereby overcoming the rigidity of conventional testing approaches. Evaluations of prominent automated software engineering frameworks—including AIDER, OpenHands, and ClaudeCode—reveal that even the strongest systems achieve only approximately 25% deliverable-quality changes, highlighting significant deficiencies in code localization and functional correctness.

Abstract

With the telecommunications field embracing zero touch management alongside novel O-RAN and AI-RAN frameworks, contemporary telecom networks now function as immensely intricate and heavily softwareized codebases. While automated software engineering (ASE) tools and Software Engineering (SWE) Agents hold the potential to alleviate the critical code generation bottleneck in this domain, their ability to navigate and modify specialized, mathematically rigorous wireless stacks like srsRAN 5G remains unverified. General-purpose coding benchmarks fail to capture the stateful logic and strict requirements of telecommunications, leaving a critical evaluation gap. In this paper, we introduce TeleSWEBench, the first commit-driven benchmark specifically designed to measure an agent's performance in the telecom domain. We mine real developer commits from the srsRAN 5G repository and distill them into structured test cases across three difficulty tiers (Easy, Medium, and Difficult). Our benchmark consists of 734 questions that are accompanied by executable unit tests. To avoid the rigidity of test cases, we further propose a hierarchical LLM as a Judge framework called TeleJudge that scores agent outputs at the file level and aggregates verdicts holistically. This follows an evaluation based on context and semantic similarity in parallel to a standard unit test-based evaluation. Using this benchmark, we evaluate AIDER, OpenHands, and the ClaudeCode frameworks, powered by state-of-the-art reasoning LLMs, including Qwen3, GPT OSS, Gemma 4, Kimi, and Qwencoder 2.5. Our two-stage evaluation reveals that models suffer from a lack of both localization accuracy and functional correctness, with the strongest ASE tools achieving up to 25% of shippable changes.

Problem

• telecommunications
• software engineering
• large language models
• code generation
• benchmark

Innovation

• commit-driven benchmark
• telecommunications software engineering
• LLM-as-a-Judge
• srsRAN 5G
• hierarchical evaluation