CUADebug: Diagnosing and Repairing Computer-Use Agent Failures

2026-07-31

AI Summary

This work addresses the persistent failures of Computer Use Agents (CUAs) in performing desktop and web tasks, which often stem from the dynamic coupling of visual perception, spatial grounding, low-level interaction, and task reasoning, making errors difficult to diagnose. To tackle this, the authors propose CUADebug, the first fine-grained error taxonomy and diagnostic framework tailored for CUAs. Leveraging a human-annotated failure benchmark from OSWorld, paired screenshots, action trajectory analysis, tool-augmented reasoning, and re-execution mechanisms, CUADebug generates structured diagnoses that specify root causes, error types, supporting evidence, and repair strategies. By transforming multimodal causal localization into actionable repair signals, the method improves joint sub-type and step-level diagnostic accuracy from 11.2% to 19.6% on the Claude-agent test set, achieves a 29.90% task completion rate with a single re-execution, and raises cumulative success rates from 12.2% to 25.86%.

Abstract

Computer-use agents (CUAs) operate real desktop and web interfaces through screenshots, mouse and keyboard actions, and stateful UI feedback, yet their failures remain difficult to diagnose and repair. Unlike text-only agents, CUA failures arise from coupled visual perception, spatial grounding, low-level interaction, task reasoning, and environment dynamics, making debugging a distinctive multimodal causal localization problem. We introduce CUADebug, a framework for diagnosing and repairing CUA failures. CUADebug includes a CUA-specific error taxonomy, CUAErrorBench, a human-annotated OSWorld failure benchmark, and CUADebugger, a tool-augmented debugger. Instead of prompting over the full trajectory once, CUADebugger actively inspects suspicious steps with paired before/after screenshots and action traces, then submits a structured diagnosis containing the root-cause step, error type, grounded evidence, and corrective strategy for re-execution. Human annotations over 204 failed trajectories show that task reasoning and control is the largest failure family (110/204), followed by perception (36), grounding/interaction (25), external/system (13), and an others category of 20 OSWorld infeasible-task cases. On the main Claude-agent split, CUADebugger improves joint subtype-and-step diagnosis from 11.2% to 19.6% with Gemini 2.5 Pro and improves consistently across debugger backbones. In single re-execution package evaluation, RCA-based conditions achieve higher task completion than history-only continuation (28.47% with machine RCA and 29.90% with our method, versus 13.89%); in continual re-execution, our method improves success from 12.2% to 25.86%, while human-oracle guidance reaches 29.21%. These results show that CUA root-cause diagnosis can provide actionable repair signals rather than merely post-hoc explanations.

Problem

• Computer-Use Agents
• Failure Diagnosis
• Multimodal Debugging
• Root-Cause Analysis
• UI Interaction

Innovation

• computer-use agents
• multimodal debugging
• root-cause analysis
• error taxonomy
• tool-augmented debugger