RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

2026-09-28

AI Summary

This study addresses the limitations of existing benchmarks in evaluating the comprehensive capabilities of coding agents to build, integrate, and diagnose heterogeneous artifacts within robotics engineering. To this end, we propose RLE-Bench, the first systematic evaluation framework designed for Robot Learning Engineer qualification assessment. This benchmark encompasses four core workflows, including interactive control, and evaluates agents’ engineering proficiency through multimodal feedback and resource constraints. Moving beyond single-metric evaluations, it employs task-specific metric aggregation to generate an RLE index, complemented by in-depth case analyses that reveal agent behavioral patterns. Experimental results provide cross-dimensional capability profiling comparisons, identifying current bottlenecks in agent performance and outlining directions for future optimization.

Abstract

Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics. Existing robotics benchmarks, however, primarily focus on the performance of individual artifacts, such as policies or controllers, offering limited coverage of coding agents'broader engineering capabilities. Real-world robotics extends beyond control: agents must build, integrate, diagnose, and improve heterogeneous artifacts under resource constraints and reason from multimodal feedback. To evaluate these broader capabilities, we introduce RLE-Bench, a benchmark of robot-learning tasks spanning four representative robotics development workflows: interactive control, policy learning, perception and estimation, and mechanical design. We use diverse task-specific metrics to evaluate the artifacts submitted by the coding agents, from the success rate the agents achieved to the policy agents trained, the harness agent built, and the mechanical structures the agent designed. We aggregate these metrics into an overall RLE Index and report workflow-specific capability profiles, enabling systematic comparison of coding agents'capabilities across multiple capability dimensions. Beyond performance ranks, we also conduct in-depth case studies examining agent behavior on representative tasks, highlighting both current capabilities and limitations, and pointing to the opportunities robotics tasks have to offer for future agent training.

Problem

• Coding Agents
• Robot Learning
• Benchmark
• Engineering Capabilities
• Multimodal Feedback

Innovation

• Coding Agents
• Robot Learning
• Benchmark
• RLE Index
• Multimodal Feedback