Exploiting Web Search Tools of AI Agents for Data Exfiltration

2025-10-10

AI Summary

This work systematically evaluates the vulnerability of large language models (LLMs) to indirect prompt injection attacks in tool-calling and retrieval-augmented generation (RAG) settings. Focusing on sensitive data leakage risks in dynamic workflows—such as web search—the study benchmarks six canonical indirect injection attack types across diverse, cross-vendor LLMs (including OpenAI, Anthropic, Meta, and open-source models) spanning multiple scales and architectures. It quantifies how model size, architectural choices, and implementation details affect defense efficacy. Results show that mainstream LLMs exhibit poor robustness: attack success rates range from 68% to 92%, with no strong correlation between parameter count and resilience. The paper’s contributions include (1) the first standardized evaluation framework tailored to tool-augmented scenarios, (2) a mitigation strategy combining adversarial training and attack-pattern attribution, and (3) an open-sourced benchmark dataset for indirect prompt injection, providing empirical foundations and methodological guidance for designing intrinsically secure LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) are now routinely used to autonomously execute complex tasks, from natural language processing to dynamic workflows like web searches. The usage of tool-calling and Retrieval Augmented Generation (RAG) allows LLMs to process and retrieve sensitive corporate data, amplifying both their functionality and vulnerability to abuse. As LLMs increasingly interact with external data sources, indirect prompt injection emerges as a critical and evolving attack vector, enabling adversaries to exploit models through manipulated inputs. Through a systematic evaluation of indirect prompt injection attacks across diverse models, we analyze how susceptible current LLMs are to such attacks, which parameters, including model size and manufacturer, specific implementations, shape their vulnerability, and which attack methods remain most effective. Our results reveal that even well-known attack patterns continue to succeed, exposing persistent weaknesses in model defenses. To address these vulnerabilities, we emphasize the need for strengthened training procedures to enhance inherent resilience, a centralized database of known attack vectors to enable proactive defense, and a unified testing framework to ensure continuous security validation. These steps are essential to push developers toward integrating security into the core design of LLMs, as our findings show that current models still fail to mitigate long-standing threats.

Problem

• Analyzing LLM vulnerability to indirect prompt injection attacks
• Evaluating how model parameters influence security weaknesses
• Identifying persistent threats despite existing defense mechanisms

Innovation

• Systematic evaluation of indirect prompt injection attacks
• Strengthened training procedures for enhanced model resilience
• Centralized database and unified testing framework for security