The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents

2026-08-06

AI Summary

This work addresses a key limitation in conventional offline imitation learning for mobile GUI agents: the inability to assess action correctness due to the omission of subsequent screen observations during training. To overcome this, the authors propose Gated Hindsight Distillation (GHD), a novel approach that leverages the next screen as a hindsight privileged signal. GHD employs a parameter-shared teacher–student architecture, wherein the teacher re-evaluates the student’s actions in light of future screen states and activates gated knowledge distillation only when the student fails but the teacher can recover correct behavior. This mechanism enhances the agent’s understanding of action–outcome causality. Evaluated on the AndroidWorld and AndroidLab benchmarks, GHD substantially outperforms GRPO, achieving significantly higher task success rates.

Abstract

GUI agents are commonly trained offline from successful interaction trajectories. Standard training decomposes each trajectory into prefix-action pairs: the agent predicts an action from the current screen and interaction history, while the subsequent observation is discarded. This removes the rationale of why an action is correct: the evidence often appears only on the subsequent screen. For example, to enable Soft Wrap, the agent should click Edit or View, but nothing reveals this until the menu opens. Without such evidence, standard imitation gives the model little chance of ever sampling and thus learning the correct reasoning. To address this issue, we propose Gated Hindsight Distillation (GHD), which uses the next screenshot as privileged information during training. A student predicts from the observable trajectory prefix, while a parameter-sharing teacher additionally observes the next screenshot and re-scores the student's on-policy responses. We apply distillation only when the student fails and the hindsight-conditioned teacher recovers the demonstrated action. GHD improves task success over GRPO on AndroidWorld and AndroidLab across two vision-language models. The code and checkpoints will be made available.

Problem

• GUI agents
• imitation learning
• hindsight information
• mobile interaction
• offline training

Innovation

• Gated Hindsight Distillation
• GUI agents
• privileged information
• knowledge distillation
• mobile interaction