Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?

2026-07-28

AI Summary

Existing desktop agent benchmarks struggle to evaluate models’ understanding of the causal state transitions induced by GUI actions. To address this gap, this work proposes Desktop-Delta Bench (DDB), the first offline, step-level benchmark focused explicitly on the relationship between GUI actions and resulting state changes, comprising 2,013 human-verified trajectories across multiple Linux applications. DDB introduces two core tasks: temporal ordering of three-frame sequences with distractors and annotation of pre- and post-action state pairs across five action categories, thereby bridging the diagnostic gap between end-to-end task success and low-level perception. Evaluation across eight model variants reveals near-random performance in temporal reasoning (exact match accuracy ≈65%) and substantially weaker recognition of complex actions like drag-and-drop compared to simple clicks (F1: 0.76 vs. 0.96), exposing systematic deficiencies such as mechanically replicating input sequences without genuine causal understanding.

Abstract

Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action- crucial for rejecting stale observations, verifying progress, and recovering from failure. This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning. We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB trajectories targets 3 failure dimensions- state verification, source tracking, and context-aware control- through 2 complementary tasks: 463 3-frame temporal-ordering instances, including 105 with a cross-trajectory decoy, and 1,550 before-after pairs labeled from 5 actions + its payload. We evaluate 8 closed and open-source model families across 32 ordering and 16 single-action settings, observing consistent gaps. Ordering remains unsaturated: best non-decoy and decoy exact-match rates are 65.1% and 65.7%. Task context improves decoy identification by 6.9 percentage points but reduces non-decoy exact match by 2.2 points; error analysis reveals systematic copying of the presented A-B-C order. Single-action results show that inferring the action family is harder than locating it: click F1 is 0.96 vs, 0.76 for drag, while recognized drags are generally localized well. DDB, thus, complements end-to-end benchmarks by filling the missing diagnostic layer between GUI grounding and final task success, enabling targeted improvements to desktop CUA verification, reliability, and recovery.

Problem

• desktop GUI transitions
• computer-use agents
• state verification
• causal understanding
• asynchronous observation

Innovation

• desktop GUI transitions
• computer-use agents
• state verification
• temporal ordering
• action grounding