WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

2026-06-08

AI Summary

This work addresses the fragmented evaluation of GUI, CLI, and coding capabilities in existing benchmarks, which fails to capture long-horizon, cross-interface coordination. The authors propose the first mixed-interface benchmark centered on real user requests, comprising 114 tasks executed in authentic Ubuntu desktop environments that require agents to integrate multimodal actions within a single execution trajectory. They introduce a novel trajectory-aware scorer that automatically evaluates performance by jointly analyzing file system states, screenshots, system logs, and action sequences, effectively detecting shortcut behaviors such as output fabrication or hard-coded responses. Experimental results reveal that even the best-performing model–runtime combinations achieve only a 41.2% pass rate, and conventional outcome-based scoring substantially overestimates actual agent capabilities.

Abstract

Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.

Problem

• computer-use agents
• hybrid interfaces
• long-horizon tasks
• cross-interface orchestration
• benchmarking

Innovation

• hybrid-interface benchmark
• long-horizon tasks
• computer-use agents
• trajectory-aware evaluation
• GUI-CLI-code orchestration