SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

2026-07-20

AI Summary

This work addresses the limitations of existing GUI interaction datasets, which suffer from insufficient coverage and short temporal horizons, making it difficult to capture rare state transitions and complex multi-step operations. To overcome this, the paper introduces the SEE framework, which constructs an explicit UI state transition graph by integrating vision-language models with UI element awareness. Guided by this graph structure, the framework employs path planning and controlled sampling to efficiently synthesize diverse, long-horizon interaction trajectories. This approach avoids unproductive loops and enables reproducible, interpretable data generation. Evaluated across multiple real-world applications, SEE generates high-quality trajectories averaging 14.8 steps in length, significantly improving agent task success rates and generalization to unseen interfaces.

Abstract

Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps. Existing pipelines often rely on costly human demonstrations or on-policy framework, which tends to over-sample common flows while missing rare transitions and complex multi-step procedures. To address this problem, we propose SEE, a two-stage data synthesis framework consisting of (i) an efficient exploration stage that builds an explicit UI transition graph over screens and elements, and (ii) a graph-based synthesis stage that composes diverse multi-step trajectories via planning and controlled sampling. This design yields reproducible and explainable data generation, while explicitly preventing spurious cycles and enabling long-horizon composition. Across multiple real-world apps, SEE produces trajectories with an average length of 14.8 steps while avoiding spurious loops, and agents fine-tuned on SEE achieve improved task success and generalization to unseen screens. We will publicly release our synthesis code and dataset.

Problem

• long-horizon
• GUI agent
• interaction trajectories
• data synthesis
• rare transitions

Innovation

• structure-aware
• UI transition graph
• trajectory synthesis
• long-horizon planning
• controlled sampling