AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

2026-06-03

AI Summary

Current evaluations of large language models are largely confined to single-turn or short-horizon tasks, failing to assess their capabilities in long-term scientific research and engineering optimization. This work proposes AutoLab—the first benchmark designed for ultra-long-horizon autonomous optimization—comprising 36 expert-crafted closed-loop tasks spanning system optimization, puzzle solving, model development, and CUDA kernel tuning, with an emphasis on temporal awareness and iterative refinement. The benchmark incorporates real-world engineering scenarios, closed-loop feedback mechanisms, and strict wall-clock time constraints, and all associated tools are open-sourced. Evaluation across 17 state-of-the-art models reveals that sustained experimentation and effective integration of feedback—not the quality of initial proposals—are key to success; Claude-Opus-4.6 achieves the best performance, while most models show limited progress due to premature termination.

Abstract

Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons. To address this gap, we introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization. AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains: system optimization, puzzle & challenge, model development, and CUDA kernel optimization. Each task begins with a correct but deliberately suboptimal baseline and challenges agents to improve it within a strict wall-clock budget. Evaluating 17 state-of-the-art models reveals the dominant predictor of success is not the quality of an agent's initial attempt, but its persistence in repeatedly benchmarking, editing, and incorporating empirical feedback. While claude-opus-4.6 exhibits strong long-horizon optimization capabilities, most frontier models, including several proprietary ones, either terminate prematurely or exhaust their budgets with minimal progress. These results underscore the importance of time awareness and persistent iteration in autonomous agents. We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.

Problem

• long-horizon
• iterative improvement
• autonomous agents
• closed-loop optimization
• benchmarking

Innovation

• long-horizon optimization
• closed-loop iteration
• autonomous agents
• empirical feedback
• benchmarking