SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

2026-07-16

AI Summary

This work addresses the critical limitation of mobile GUI agents—their inability to anticipate risks prior to action execution, which can lead to irreversible consequences from a single erroneous operation. To mitigate this, the authors propose SeerGuard, a novel safety framework featuring the first Safety-Augmented World Model (SAWM). SAWM unifies semantic next-state prediction, instruction-level filtering, and action-level risk assessment to enable proactive risk intervention. Built upon multi-task learning and integrated into Qwen3-VL-8B-Instruct, SAWM significantly enhances both safety and utility, improving the safety-utility score from 0.191 to 0.596 while reducing the risk-cost score from 0.347 to 0.130.

Abstract

Mobile graphical user interface (GUI) agents have demonstrated remarkable capabilities in automating complex tasks, yet they introduce critical safety risks where a single erroneous action can lead to irreversible consequences. Existing safety mechanisms are primarily reactive, lacking the ability to assess risks before execution. In this paper, we introduce SeerGuard, a consequence-aware safety framework designed to mitigate these risks through pre-execution instruction-level screening and action-level risk assessment. Specifically, the action-level assessment analyzes agent-proposed actions within current GUI states, anticipating likely outcomes to identify risks before they are executed. To enable these capabilities, we construct a unified safety-augmented world model (SAWM) via multi-task learning, integrating semantic next-state prediction with safety risk assessment. Extensive experiments demonstrate that SeerGuard generalizes effectively across diverse mobile GUI agents. On Qwen3-VL-8B-Instruct, it increases the safety-utility score from $0.191$ to $0.596$ at $ω=0.8$ and reduces the risk-cost score from $0.347$ to $0.130$ at $α=0.8$. Further analyses on our SAWM validate the effectiveness of the instruction-level screening, alongside the capability of action risk assessment and next-state prediction.

Problem

• mobile GUI agents
• safety risks
• pre-execution risk assessment
• irreversible consequences
• reactive safety mechanisms

Innovation

• world model
• safety framework
• risk assessment
• mobile GUI agents
• multi-task learning