AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents

2026-05-25

AI Summary

This work addresses critical security vulnerabilities in large language model (LLM) agents arising from the shared generative channel used for instructions, retrieved content, and tool observations, which renders them susceptible to unauthorized inputs—leading to prompt injection, privacy leakage, and tool misuse. The paper introduces the first formal security framework grounded in the principle of “intent-to-execution non-interference,” which formalizes application policies as projection operations over authorized observations and capabilities. It distinguishes between prompt annotations and enforcement mechanisms, and establishes a measurable security evaluation paradigm centered on channel closure. Empirical validation across three adversarial game tasks on Qwen3-0.6B and Qwen3-1.7B demonstrates that safety cannot be ensured by prompt-based descriptions alone; only execution-layer enforced channel closure effectively mitigates risks, preserving instruction integrity, retrieval confidentiality, and capability fidelity.

Abstract

LLM agents process trusted instructions, retrieved records, and tool observations through a common generative channel. This conflates data flow with authority: an untrusted string can affect a secret-bearing response or an action proposal even when no application policy authorizes that influence. We introduce AgentSecBench as an empirical instantiation of a formal security framework for this problem. The framework defines three games-instruction-integrity, retrieval-confidentiality, and capability-integrity-under a common notion of intent-to-execution noninterference with permitted leakage. It represents an application policy as a projection onto authorized observations and capabilities, distinguishes prompt annotations from enforcing projections, and measures both adversarial advantage and whether a defense closes the relevant model-visible channel before generation. The exact-marker experiments are intentionally one observable instantiation of the games rather than a complete semantic security claim: they test disclosure and forbidden-action distinguishers with unambiguous ground truth. We evaluate six defense classes with Qwen3-0.6B and Qwen3-1.7B on paired adversarial and benign-control executions. The measurements show when risk reduction follows channel closure and when a model-visible adversarial capability remains exploitable. The result is a security-oriented evaluation method: prompt text can describe a boundary, whereas provenance projection, capability restriction, and output validation can enforce one.

Problem

• prompt injection
• privacy leakage
• tool-use integrity
• LLM agents
• security evaluation

Innovation

• AgentSecBench
• prompt injection
• noninterference
• capability integrity
• provenance projection