StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

2026-08-06

AI Summary

This work addresses the inefficiency of existing mobile GUI agents, which rely on costly and unstructured multimodal reasoning for action reflection, hindering their ability to perform long-horizon tasks effectively. The paper introduces the first formalization of GUI action reflection as a structured prediction task grounded in explicit transition specifications and visual evidence, and proposes a dedicated, locally deployable reflection model (8B parameters) that achieves high accuracy at low cost. Through supervised fine-tuning, teacher-student distillation, and preference- and reward-driven refinement, the model attains an 82.16% state-transition accuracy on the AndroidWorld offline benchmark—surpassing zero-shot GPT-5.2 by 11.83 percentage points. In online experiments, it outperforms baselines in three out of four agent task categories while significantly reducing API invocation costs.

Abstract

Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions. We propose StepReflect, which formulates per-step GUI reflection as supervised structured prediction conditioned on explicit transition specifications and paired visual evidence. StepReflect is trained through a staged pipeline combining supervised fine-tuning, teacher-student distillation, and preference- and reward-based refinement. Offline, the resulting 8B model achieves 82.16% transition-level accuracy on AndroidWorld, exceeding zero-shot GPT-5.2 by 11.83 percentage points under the same structured input. Online, across M3A, Agent-SAMA, MAI-UI-8B, and Seed-2.0-Pro, StepReflect achieves higher task success in three of four agent configurations and remains within one successful task of the GPT-5.2 Reflection Agent in the fourth. It also reduces paid API charges relative to GPT-based reflection in all four configurations. These results establish StepReflect as a practical, locally deployable alternative to repeated frontier-model reflection for long-horizon mobile GUI agents.

Problem

• mobile GUI agents
• action reflection
• structured prediction
• UI transition
• long-horizon execution

Innovation

• structured prediction
• GUI reflection
• mobile GUI agents
• teacher-student distillation
• state transition modeling