Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion

2026-07-31

AI Summary

Current alignment evaluations for GUI agents predominantly rely on single-turn explicit refusals, which inadequately capture true robustness under implicit intentions in dynamic, multi-turn interactions. This work advances the perspective that “alignment is local” and introduces a user-side, screen-anchored four-turn persuasion-chain diagnostic framework. By integrating natural language persuasion chains with screen semantic grounding—without requiring environment injection—it assesses the alignment stability of state-of-the-art GUI agents and incorporates a lightweight prompt-layer defense mechanism. Experiments show that single-turn defenses reduce attack success rates by up to 40 percentage points with negligible over-refusal; however, under a four-turn escalating persuasion chain, all defended models exhibit a rebound in attack success rates of approximately 20 percentage points, revealing significant vulnerability to subtle, incremental requests.

Abstract

Trustworthy deployment of GUI agents in ubiquitous computing settings requires alignment that survives dynamic interaction and precise threat conditions, not just single-turn refusal of explicit harmful requests. We argue that prompt-level alignment, the dominant lightweight defense in current mobile agents, is a local phenomenon: it works reliably only in the narrow evaluation slice where it is typically measured, namely single-turn, explicitly-verbalized intent, and degrades systematically along two axes that any real user can traverse. Using a paired diagnostic on three frontier GUI agents, screen-grounded, user-side persuasion, with no environment injection, we show that a one-line guardrail achieves large single-shot ASR reductions, up to roughly 40 points, at near-zero over-refusal cost. Nevertheless, moving from independent probes to four-turn escalation chains raises guarded ASR by approximately 20 points on every model. Relative to the neutral baselines, this increase reflects substantial guardrail erosion for Qwen but a largely defense-orthogonal dynamic risk for Claude and GPT. The sign of the salience gap flips under the guardrail: concealed requests are not systematically more successful than explicit ones without a guardrail, but are more successful with one, indicating that the defense engages primarily when intent is named. Static single-turn ASR therefore overstates deployed robustness by a systematic and predictable margin.

Problem

• alignment
• GUI agents
• user-side persuasion
• dynamic interaction
• robustness

Innovation

• local alignment
• paired diagnostic
• user-side persuasion
• guardrail erosion
• salience gap