Mind the Web: The Security of Web Use Agents

2025-06-08

AI Summary

This work identifies a novel security threat posed by web-use agents—AI systems endowed with high-privilege capabilities such as DOM manipulation, JavaScript execution, and authenticated session access. Attackers exploit malicious web content (e.g., comments, ads) and introduce “task-aligned injection,” a technique that disguises adversarial instructions as legitimate task directives, leveraging inherent limitations in LLM contextual reasoning to divert agents from their intended goals. The study is the first to systematically characterize context-aware vulnerabilities in such agents, proposing a comprehensive security evaluation framework integrating multi-agent penetration testing, execution sandboxing, and task-intent modeling. We reproduce nine high-severity attack classes—including unauthorized camera activation and credential exfiltration—across four state-of-the-art web-use agents, achieving 80–100% success rates across diverse LLM backends. Finally, we propose deployable countermeasures: runtime supervision, task-consistency constraints, and reasoning-augmented inference mechanisms.

Technology Category

• Multiagent Systems: Adversarial Agents
• Application Domains: Security
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Web-use agents are rapidly being deployed to automate complex web tasks, operating with extensive browser capabilities including multi-tab navigation, DOM manipulation, JavaScript execution and authenticated session access. However, these powerful capabilities create a critical and previously unexplored attack surface. This paper demonstrates how attackers can exploit web-use agents' high-privilege capabilities by embedding malicious content in web pages such as comments, reviews, or advertisements that agents encounter during legitimate browsing tasks. In addition, we introduce the task-aligned injection technique that frame malicious commands as helpful task guidance rather than obvious attacks. This technique exploiting fundamental limitations in LLMs' contextual reasoning: agents struggle in maintaining coherent contextual awareness and fail to detect when seemingly helpful web content contains steering attempts that deviate from their original task goal. Through systematic evaluation of four popular agents (OpenAI Operator, Browser Use, Do Browser, OpenOperator), we demonstrate nine payload types that compromise confidentiality, integrity, and availability, including unauthorized camera activation, user impersonation, local file exfiltration, password leakage, and denial of service, with validation across multiple LLMs achieving success rates of 80%-100%. These payloads succeed across agents with built-in safety mechanisms, requiring only the ability to post content on public websites, creating unprecedented risks given the ease of exploitation combined with agents' high-privilege access. To address this attack, we propose comprehensive mitigation strategies including oversight mechanisms, execution constraints, and task-aware reasoning techniques, providing practical directions for secure development and deployment.

Problem

• Exploiting web-use agents' high-privilege capabilities via malicious web content
• Task-aligned injection disguising attacks as helpful task guidance
• Mitigating risks of unauthorized access and data leakage in web agents

Innovation

• Exploiting web agents' high-privilege capabilities maliciously
• Introducing task-aligned injection for covert attacks
• Proposing oversight mechanisms for secure deployment