Scaling GUI Agents with Visual State Transitions

2026-07-27

AI Summary

This work addresses the limited generalization of GUI agents in cross-platform tasks by proposing State Transition Pretraining (STP), which introduces state transition modeling as a novel dimension for GUI understanding. STP jointly optimizes inverse and forward dynamics to continually pretrain a unified multimodal model on large-scale visual state transition data, thereby learning action-anchored visual representations and a dynamic world model of GUIs. When combined with instruction fine-tuning, STP significantly outperforms baseline methods that rely solely on trajectory fine-tuning across multiple benchmarks—including AgentNetBench, AndroidControl, and GUIOdyssey—and demonstrates consistent performance gains with increasing data scale, confirming its effectiveness and scalability.

Abstract

We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions). This optimization equips the model with better action-grounded visual representations and an internal world model of GUI dynamics. When subsequently fine-tuned on trajectories with task instructions, our STP-trained models consistently outperform baselines trained solely via direct trajectory fine-tuning across agent benchmarks in both desktop and mobile GUI scenarios (AgentNetBench, AndroidControl, and GUIOdyssey). Further empirical studies show that joint dynamics optimization yields stable improvements over single-objective training, and downstream performance scales steadily with the volume of transition data.

Problem

• GUI agents
• visual state transitions
• scaling
• multimodal learning
• task generalization

Innovation

• State Transition Pretraining
• inverse dynamics
• forward dynamics
• multimodal GUI agents
• visual state transitions