Assessing Automated Prompt Injection Attacks in Agentic Environments

2026-06-09

AI Summary

This study addresses the critical security threat posed by indirect prompt injection attacks against large language model (LLM) agents that interact with untrusted external data, an area lacking systematic investigation in realistic agent environments. Within the AgentDojo framework, we present the first adaptation of both white-box (GCG) and black-box (TAP) attack methodologies to the agent setting, conducting comprehensive evaluations across 80 diverse tasks and multiple models. Our findings reveal that black-box optimization significantly outperforms gradient-based approaches under practical computational constraints. While universal attacks demonstrate strong transferability to unseen tasks and out-of-domain scenarios, attacks generated by smaller models fail to generalize effectively to state-of-the-art models such as GPT-5. This work uncovers key challenges in agent security and establishes an empirical foundation for developing future defense mechanisms.

Abstract

Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.

Problem

• prompt injection
• LLM agents
• automated attacks
• agentic environments
• adversarial robustness

Innovation

• automated prompt injection
• LLM agents
• black-box optimization
• attack transferability
• AgentDojo