PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

2026-09-28

AI Summary

This study addresses the inefficiency, high cost, and fragility arising from frame-by-frame visual perception in mobile GUI agents. It proposes an acceleration framework that compiles application navigation into callable commands. Specifically, the method constructs semantically annotated maps via offline exploratory distillation, deterministically replays execution sequences during online inference, and falls back to a vision-language model (VLM) interpreter upon encountering anomalies—all without requiring internal APIs or model training. Experiments on the AndroidLab benchmark demonstrate that this framework substantially improves task success rates while significantly reducing step counts and token consumption. Furthermore, it successfully transfers to the AndroidWorld M3A agent, achieving efficient generalization from static, repetitive navigation to universal task support.

Abstract

Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.

Problem

• Mobile GUI agents
• Vision-language model
• Navigation efficiency
• Perception-action loop
• Token consumption

Innovation

• Mobile GUI Agents
• Vision-Language Model
• Command Compilation
• Navigation Distillation
• Deterministic Execution