Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees

2026-07-30

AI Summary

This work addresses the challenge that GUI agents often miss transient interface events due to autoregressive decoding latency. To overcome this, the authors propose Adaptive Anticipatory Policy Trees (AAPT), which leverage a frozen multimodal model during idle periods to precompute policy trees containing observable conditions, pre-authorized actions, and branch expiration times. Upon event occurrence, a lightweight observer rapidly matches the current context to a precomputed branch and executes the corresponding action immediately, bypassing real-time generation. This approach uniquely integrates policy precompilation with lightweight runtime routing, eliminating decision latency without modifying the underlying model and revealing that performance bottlenecks stem from branch routing rather than tree planning. Experiments demonstrate a significant improvement in success rate—from 0.50 to 0.79 (p = 1.8 × 10⁻³)—within critical decision windows, with zero erroneous actions and consistent replication across 126 independent trials (p = 4.9 × 10⁻¹³).

Abstract

Computer-use agents often fail on transient GUI events because they produce the correct action only after the relevant window has already closed. We identify the main cause as expensive autoregressive decoding on the decision-time critical path. We propose Adaptive Anticipatory Policy Trees (AAPT), which eliminates this delay without modifying the underlying model. During idle screen periods, the same frozen multimodal model constructs a bounded conditional policy tree with observable guards, pre-authorized actions, and branch-specific deadlines. The tree is sized to cover the model's own decoding latency. When an event occurs, a lightweight observer matches change-gated frames to a prepared branch and immediately executes the corresponding action without generating new text. In paired trials with pre-registered endpoints and exact McNemar tests, AAPT improves the success rate from 0.50 to 0.79 within a contested decision window ($p=1.8\times10^{-3}$), while producing no incorrect actions. Both open-loop and predict-and-replan baselines achieve zero success because they still decode during execution. A preparation-time sweep shows that the gain emerges where the latency-based tree-sizing rule predicts, and ablations reveal three key requirements: fast observer decoding, valid tree planning, and accurate branch routing. A pre-registered oracle probe rejects our initial hypothesis and instead points to branch routing as the causal bottleneck. We further reproduce the effect on an independent general-purpose multimodal model over 126 paired trials ($p=4.9\times10^{-13}$). On an external benchmark, AAPT matches the overall performance of a reactive baseline, although the two methods exhibit complementary strengths. Together, these results suggest that AAPT performs best when candidate actions can be enumerated in advance, whereas reactive execution remains stronger when they cannot.

Problem

• GUI agents
• transient events
• decision-time latency
• action timing
• response delay

Innovation

• Adaptive Anticipatory Policy Trees
• decision-time critical path
• pre-compiled policy trees
• multimodal GUI agents
• latency-aware planning