CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents

2026-06-10

AI Summary

This work addresses the incidental visual privacy risks in mobile GUI agents, where screenshots captured during task execution may inadvertently expose sensitive, task-irrelevant information such as contacts or messages. To mitigate this, the authors propose a context-aware privacy protection layer deployed on-device that, prior to uploading screenshots to a remote multimodal agent, selectively retains only task-essential content while obscuring irrelevant elements. This approach integrates task semantic understanding, UI element parsing, and privacy prior modeling, treating screenshot upload as a critical privacy decision point at the device–cloud boundary. It overcomes limitations of conventional full-screen masking or text anonymization. Evaluated across 28 privacy-sensitive tasks, the method reduces weighted privacy leakage from 0.766 to 0.268 while maintaining high task success rates, with effectiveness and practicality validated on AndroidWorld.

Abstract

Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user, but this capability also turns every screen observation into a privacy boundary. During normal task execution, screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context that is unrelated to the user's request. We call this problem incidental visual privacy exposure. It is difficult to address with existing defenses: text anonymization misses many visual and inferential cues, while generic privacy masking can remove the evidence and controls that a GUI agent needs to complete the task. This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents. CAPED is designed as a phone-side protection layer: before screenshots are released to a remote multimodal agent, it extracts task requirements, uses screen context as a privacy prior, parses visible UI elements, and selectively exposes only content needed for the current task while masking incidental private content. We evaluate CAPED on AndroidWorld for broad task utility and with a controlled 28-task seeded privacy evaluation used as a measurement instrument for trajectory-level incidental leakage. In this seeded evaluation, Full CAPED reduces success-conditioned weighted seeded leakage from 0.766 under raw screenshots to 0.268 while preserving high task utility. A broader AndroidWorld run shows a remaining prototype-level utility cost, but the results support the central claim that screenshot upload should be treated as an explicit device--cloud boundary decision, governed by task-driven selective exposure rather than all-or-nothing screen sharing.

Problem

• incidental visual privacy exposure
• mobile GUI agents
• screenshot-based privacy
• context-aware privacy
• privacy leakage

Innovation

• context-aware privacy
• mobile GUI agents
• selective exposure
• incidental visual privacy exposure
• privacy-preserving screenshot