Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

2025-06-10

AI Summary

Existing mobile agents exhibit limited generalization to atomic tasks, struggling with real-world composite tasks that involve multi-step reasoning, cross-interface navigation, and context-dependent execution. Method: We introduce UI-NEXUS—the first benchmark explicitly designed for generalization from atomic to composite tasks—and propose AGENT-NEXUS, a lightweight scheduling system. It features dynamic subtask decomposition and state-aware scheduling, enabling seamless composition of atomic capabilities into composite reasoning without model retraining. The system integrates a multimodal large language model–driven architecture, a controllable local UI simulation environment, and an online evaluation framework supporting both English- and Chinese-language mobile applications. Contribution/Results: We systematically define and evaluate three composite operation types: simple concatenation, context transfer, and deep exploration. Experiments show AGENT-NEXUS improves task success rates by 24%–40% on UI-NEXUS, significantly mitigating under-execution, over-execution, and attention drift, while incurring negligible inference overhead.

Technology Category

• Multiagent Systems: Agent Communication
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Autonomous agents powered by multimodal large language models have been developed to facilitate task execution on mobile devices. However, prior work has predominantly focused on atomic tasks -- such as shot-chain execution tasks and single-screen grounding tasks -- while overlooking the generalization to compositional tasks, which are indispensable for real-world applications. This work introduces UI-NEXUS, a comprehensive benchmark designed to evaluate mobile agents on three categories of compositional operations: Simple Concatenation, Context Transition, and Deep Dive. UI-NEXUS supports interactive evaluation in 20 fully controllable local utility app environments, as well as 30 online Chinese and English service apps. It comprises 100 interactive task templates with an average optimal step count of 14.05. Experimental results across a range of mobile agents with agentic workflow or agent-as-a-model show that UI-NEXUS presents significant challenges. Specifically, existing agents generally struggle to balance performance and efficiency, exhibiting representative failure modes such as under-execution, over-execution, and attention drift, causing visible atomic-to-compositional generalization gap. Inspired by these findings, we propose AGENT-NEXUS, a lightweight and efficient scheduling system to tackle compositional mobile tasks. AGENT-NEXUS extrapolates the abilities of existing mobile agents by dynamically decomposing long-horizon tasks to a series of self-contained atomic subtasks. AGENT-NEXUS achieves 24% to 40% task success rate improvement for existing mobile agents on compositional operation tasks within the UI-NEXUS benchmark without significantly sacrificing inference overhead. The demo video, dataset, and code are available on the project page at https://ui-nexus.github.io.

Problem

• Evaluates mobile agents' generalization from atomic to compositional tasks
• Identifies performance-efficiency imbalance in existing mobile agents
• Proposes scheduling system to improve compositional task success rates

Innovation

• Introduces UI-NEXUS benchmark for compositional tasks
• Proposes AGENT-NEXUS lightweight scheduling system
• Dynamically decomposes tasks into atomic subtasks