AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents

2026-08-06

AI Summary

Current mobile GUI agents are hindered by scarce real-world interaction trajectories, high costs of simulation environments, unstable world models, limited modality coverage, and inconsistent action-state transitions. This work proposes the first incremental code-based world model incorporating action-transition constraints, achieving high-fidelity simulation by predicting structurally consistent and executable GUI code updates rather than unconstrained images or text. The approach employs a two-stage HTML generation mechanism: it retrieves structural references from the current screen and action to produce executable HTML embedded with visual assets. Integrated with Code2World evaluation and reinforcement learning fine-tuning, the model enables closed-loop policy optimization. Evaluated on CMGUIBench-500, it significantly outperforms baselines in fidelity; the trained AppDeltaAgent achieves state-of-the-art performance on AndroidLens and demonstrates continual improvement on MobileGym and MobileWorld, supporting test-time policy adaptation without real-environment interactions.

Abstract

Mobile GUI agents can operate apps through pixel perception and touch actions, making them a promising interface for collecting and improving long-horizon mobile interaction policies. However, real trajectories are difficult to obtain for sensitive apps and privacy-critical operations. At the same time, existing simulated environments are costly to scale up, and GUI world models still suffer from unstable generation, limited modality coverage, and inconsistent action-transition logic. To address these limitations, we propose AppDeltaWorld, a transition-grounded delta code world model that predicts the next GUI as a reachable code update rather than as an unconstrained image or text description. AppDeltaWorld retrieves app-specific Level-1 HTML references under an action-transition constraint, generates Level-2 executable HTML conditioned on the current screen, action, predicted next-screen text, and retrieved structure, and inserts generated visual assets into image slots before browser rendering. As a world model, AppDeltaWorld achieves the highest fidelity on CMGUIBench-500 under Code2World evaluation, with clear gains in structural layout and UI element reconstruction over image-only and code-only baselines. As a training environment, AppDeltaWorld supports filtered closed-loop SFT data construction that, when combined with public supervision, enables AppDeltaAgent to achieve state-of-the-art performance on AndroidLens and consistent gains on MobileGym and MobileWorld. Moreover, world-model-based test-time reinforcement learning enables policy adaptation and shows further improvements without additional interaction with real apps.

Problem

• mobile GUI agents
• world model
• action-transition logic
• privacy-critical operations
• simulated environments

Innovation

• delta code world model
• transition-grounded generation
• mobile GUI agent
• structured HTML rendering
• world-model-based RL