Problem
Research questions and friction points this paper is trying to address.
long-horizon tasks
GUI agents
professional workflows
domain-specific software
autonomous computer use
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
🤖 AI Summary
Current AI agent evaluation benchmarks lack support for long-horizon, high-value graphical user interface (GUI) workflows in professional domains. This work proposes the first benchmark specifically designed for long-horizon GUI tasks within professional software environments, modeling real-world workflows to systematically assess agents’ capabilities in instruction following, multi-stage execution, and procedural consistency. Experimental results reveal that even the strongest existing models achieve a success rate barely exceeding 30%, highlighting significant deficiencies in error propagation control, goal maintenance, and complex workflow execution. These findings underscore the benchmark’s critical role in exposing the limitations of current agents and advancing the development of professional-grade GUI-capable AI systems.
📝 Abstract
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple applications, and short-horizon tasks, leaving it largely unknown whether modern agents can follow user instructions to autonomously operate domain-specific professional software and accomplish economically valuable work in an end-to-end manner. To bridge this gap, we introduce Workflow-GYM, a benchmark for long-horizon GUI tasks centered on professional domains and specialized software environments. Through extensive experiments on state-of-the-art models, we find that even the strongest models achieve only slightly above 30% success rates, highlighting that professional long-horizon GUI workflows remain highly challenging for current GUI agents. Further analysis reveals that current agents struggle to maintain long-horizon workflow consistency, frequently exhibiting workflow stage omission, error propagation, objective drift, and insufficient understanding of professional software environments. Our findings provide important insights into the limitations of current agent systems and suggest key directions for the next generation of GUI-agent research.
Innovation
Methods, ideas, or system contributions that make the work stand out.
long-horizon GUI tasks
professional software workflows
agent evaluation benchmark
domain-specific automation
Workflow-GYM
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
