cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

2026-09-30

AI Summary

This study addresses the irreproducibility of speed evaluations in computer-use agent benchmarks caused by infrastructural discrepancies. To this end, it proposes a standardized evaluation framework that enables seamless cross-benchmark assessment by unifying virtual machine environments, standardizing agent interfaces, and introducing task set distillation techniques. The findings reveal that the effects of inference overhead and environment latency on execution speed are counterintuitive, and that no existing model achieves comprehensive optimality. Furthermore, the distilled task sets significantly enhance evaluation efficiency without compromising statistical power.

Abstract

Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA benchmarks currently face a reproducibility crisis. Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA benchmarks without degrading overall statistical power, allowing for more efficient benchmarking and comparison. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at https://cuaspeedrun.com.

Problem

• Computer-Use Agents
• Benchmarking
• Execution Speed
• Reproducibility
• Standardization

Innovation

• Computer-Use Agents
• Standardized Benchmarking
• Execution Speed
• Agent Interface
• Evaluation Efficiency