Unlocking Smarter Device Control: Foresighted Planning with a World Model-Driven Code Execution Approach

2025-05-22

AI Summary

In autonomous mobile device control, limited visual observations and reactive policy execution lead to suboptimal decision-making. Method: This paper proposes a world-model-driven proactive planning framework that integrates visual encoding, large language model (LLM)-enhanced task-oriented world modeling, iterative symbolic reasoning, and Python code generation—enabling an end-to-end pipeline from natural language understanding to structured planning and executable actions. Contribution/Results: Its core innovation is the first-ever *iteratively updatable task-oriented world model*, enabling multi-step collaborative decision-making and online model refinement. In simulation, the framework achieves a 44.4% higher task success rate than state-of-the-art methods. Real-device evaluation demonstrates strong robustness and generalization in complex cross-app tasks, including information retrieval and form filling.

Technology Category

Planning, Routing, and Scheduling: Planning with Language ModelsHumans and AI: Human-Aware Planning and Behavior PredictionMachine Learning: Large Multimodal Models (LMMs)

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The automatic control of mobile devices is essential for efficiently performing complex tasks that involve multiple sequential steps. However, these tasks pose significant challenges due to the limited environmental information available at each step, primarily through visual observations. As a result, current approaches, which typically rely on reactive policies, focus solely on immediate observations and often lead to suboptimal decision-making. To address this problem, we propose extbf{Foresighted Planning with World Model-Driven Code Execution (FPWC)},a framework that prioritizes natural language understanding and structured reasoning to enhance the agent's global understanding of the environment by developing a task-oriented, refinable emph{world model} at the outset of the task. Foresighted actions are subsequently generated through iterative planning within this world model, executed in the form of executable code. Extensive experiments conducted in simulated environments and on real mobile devices demonstrate that our method outperforms previous approaches, particularly achieving a 44.4% relative improvement in task success rate compared to the state-of-the-art in the simulated environment. Code and demo are provided in the supplementary material.

Problem

• Enhancing mobile device control with foresighted planning
• Overcoming limited environmental information in sequential tasks
• Improving decision-making via world model-driven code execution

Innovation

• World Model-Driven Code Execution for foresighted planning
• Natural language understanding enhances environmental comprehension
• Iterative planning within a refinable world model