DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration

2026-06-01

AI Summary

Existing desktop GUI benchmarks oversimplify real-world, long-horizon workflows from professional software into short tasks and assume instructions are provided all at once, failing to capture the complexities of human–agent collaboration. To address this, this work introduces DeskCraft, a benchmark centered on creative and engineering applications involving tasks exceeding 50 steps. It establishes the first comprehensive human–agent collaboration protocol that explicitly incorporates active clarification, user interruptions, and feedback loops, and curates 538 structured tasks grounded in real-world usage of design, video, audio, and 3D software. Experimental evaluation across 18 agents reveals that even the strongest model, GPT-5.4, achieves only 31.6% and 27.6% success rates on standard and interactive tasks, respectively, highlighting significant limitations in long-horizon execution and proactive collaboration. All code and data will be publicly released.

Abstract

Real-world professional desktop workflows in specialized creative and engineering software unfold over long horizons and often require human-in-the-loop coordination, where agents proactively seek necessary information and users provide additional instructions, clarifications, feedback, or corrections as the task progresses. Yet existing desktop GUI benchmarks mostly reduce this setting to short, simplified tasks with all user instructions provided upfront. To address this issue, we introduce DeskCraft, a desktop GUI benchmark targeting long horizon creative and engineering workflows and proactive human-agent collaboration. DeskCraft organizes tasks into a multilevel difficulty taxonomy, with long horizon tasks requiring over 50 execution steps, and covers professional creative software across design, video, audio, and 3D creation. Furthermore, DeskCraft formalizes human-agent collaboration into an interaction protocol covering mid-turn and post-turn exchanges. Mid-turn interaction captures both agent-initiated clarification under uncertainty and user-initiated interruption during execution, while post-turn interaction accommodates user-driven feedback after the agent signals completion, together spanning the full space of realistic collaboration patterns. We evaluate 18 proprietary and open source agents on 538 tasks and find that GPT-5.4 reaches 31.6% on standard tasks and 27.6% on interactive tasks. Further analyses reveal persistent failures in long horizon workflow delivery and proactive clarification. We will open-source all evaluation codes, tasks, and data at https://github.com/mrwwk/DeskCraft.

Problem

• desktop GUI benchmark
• long-horizon workflows
• human-in-the-loop collaboration
• professional creative software
• agent evaluation

Innovation

• desktop GUI benchmark
• long-horizon workflows
• human-in-the-loop collaboration
• interactive task protocol
• professional creative software