GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

2026-09-28

AI Summary

Existing GUI agent evaluations overlook environmental structure, rendering systematic failures invisible and obscuring long-tail issues. This work proposes GUITAR, a state-centric framework that leverages State Transition Graphs (STGs) to map visually diverse screens onto shared functional states, thereby transcending the limitations of conventional step accuracy metrics to enable cross-screen systematic failure localization and structured diagnosis. Experiments on the AndroidControl and Mind2Web datasets reveal that 60.4% of failures concentrate within merely 20% of states. Furthermore, guided optimization based on bottleneck states yields a 2.8% improvement in success rate, validating the effectiveness of structure-aware evaluation.

Abstract

Understanding where and why Graphical User Interface (GUI) agents fail is essential for building more reliable systems, yet current evaluation relies on step accuracy, a metric that treats each screen independently and overlooks the underlying structure of GUI environments. This leads to two critical blind spots: (1) functionally equivalent screens are evaluated in isolation, obscuring systematic failure patterns across shared screens; and (2) the long-tailed GUI distribution renders failures on rare but critical screens invisible under standard metrics. To address these issues, we propose \textbf{GUITAR}, a state-centric diagnostic framework that performs structured failure analysis over both states and transitions, using a State Transition Graph (STG) by mapping visually diverse screens to shared functional states. Across 8 agents and 6 tasks from AndroidControl and Mind2Web, GUITAR reveals that 60.4\% of failures occur in 20\% of states, localizing errors to a small set of bottlenecks. Bottleneck-targeted guidance improves SR by 2.8\% and retains a 1.88\% average gain across 7 agents under three-fold trajectory-held-out evaluation with fully automatic STGs. These findings demonstrate the diagnostic and actionable value of structure-aware evaluation within the evaluated mobile and web tasks. Code is available at https://github.com/sqzhang-lazy/GUITAR

Problem

• GUI agents
• failure diagnosis
• evaluation metrics
• state transitions
• long-tailed distribution

Innovation

• GUI Agents
• State Transition Graph
• Failure Diagnosis
• Bottleneck Localization
• Structure-aware Evaluation