EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

2026-07-18

AI Summary

This work addresses the limitation of current GUI agent evaluations, which predominantly rely on end-to-end success rates and fail to disentangle agents’ understanding of interface state transitions. To this end, the authors propose EvoGUI-Bench, the first diagnostic benchmark that requires no additional annotations and automatically generates three types of visual question-answering tasks—temporal ordering, inverse action/value prediction, and contrastive single-step successor discrimination—from interaction trajectories to independently assess state evolution awareness. Built upon Mind2Web and WebLINX, the benchmark comprises 3,000 samples across 120 domains and enables zero-shot evaluation using vision-language models. Evaluation across 28 model configurations reveals that even the best-performing model, EvoGain, achieves only 60.4% accuracy, highlighting a significant gap in current methods’ ability to comprehend state transitions.

Abstract

GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery. We introduce EvoGUI, a diagnostic framework that converts normalized GUI trajectories into three complementary visual question answering probes: temporal ordering, inverse action/value prediction, and contrastive one-step successor discrimination. Their labels are derived from trajectory order and logged actions, requiring no additional task-label annotation after trajectory normalization. We instantiate EvoGUI-Bench from Mind2Web and WebLINX, yielding 3,000 instances across 120 domains, and evaluate 28 vision-language model configurations zero-shot. The strongest model reaches only 60.4 EvoGain, while model scale and GUI specialization do not reliably predict performance. These results establish EvoGUI-Bench as a scalable diagnostic complement to end-to-end GUI-agent evaluation while exposing substantial headroom in state-transition understanding. The source code is publicly available at https://github.com/Yyhhh6/EvoGUI.

Problem

• GUI state-transition understanding
• diagnostic benchmark
• visual question answering
• trajectory analysis
• vision-language models

Innovation

• GUI state-transition understanding
• diagnostic benchmark
• visual question answering
• zero-shot evaluation
• trajectory normalization