OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

2024-12-27

AI Summary

GUI agent training faces critical challenges including scarcity of high-quality trajectory data, high human annotation costs, and limited diversity and realism of synthetic data. Method: This paper proposes an inverse task synthesis paradigm: first, collecting raw trajectories via a vision-language model (VLM)-driven, environment-aware, stepwise interaction; second, inferring task definitions in reverse using a learnable, trajectory-level reward model. Unlike conventional “define-then-execute” synthesis, this approach enables end-to-end, autonomous generation of trajectories that are high-quality, diverse, and grounded in real-world GUI environments. Contribution/Results: Evaluated on online benchmarks, the synthesized data significantly improves GUI agent task completion rates. Across all quality and diversity metrics—including task fidelity, action variety, and environmental consistency—the proposed method consistently outperforms existing synthetic data generation approaches.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. Despite their utility in advancing digital automation, a critical bottleneck persists: collecting high-quality trajectory data for training. Common practices for collecting such data rely on human supervision or synthetic data generation through executing pre-defined tasks, which are either resource-intensive or unable to guarantee data quality. Moreover, these methods suffer from limited data diversity and significant gaps between synthetic data and real-world environments. To address these challenges, we propose OS-Genesis, a novel GUI data synthesis pipeline that reverses the conventional trajectory collection process. Instead of relying on pre-defined tasks, OS-Genesis enables agents first to perceive environments and perform step-wise interactions, then retrospectively derive high-quality tasks to enable trajectory-level exploration. A trajectory reward model is then employed to ensure the quality of the generated trajectories. We demonstrate that training GUI agents with OS-Genesis significantly improves their performance on highly challenging online benchmarks. In-depth analysis further validates OS-Genesis's efficiency and its superior data quality and diversity compared to existing synthesis methods. Our codes, data, and checkpoints are available at href{https://qiushisun.github.io/OS-Genesis-Home/}{OS-Genesis Homepage}.

Problem

• Automated Data Collection
• GUI Agents Training
• High-Quality Data

Innovation

• OS-Genesis
• Reverse Task Synthesis
• GUI Agent Optimization