ContainmentBench: Trace-Based Evaluation of Post-Injection Containment in Tool-Using LLM Agents

2026-07-27

AI Summary

This work addresses a critical gap in existing prompt injection evaluations, which focus solely on final outcomes while overlooking post-attack trajectory divergence and degradation of authorized functionalities. To remedy this, the authors introduce ContainmentBench, a novel benchmark that pioneers a trajectory-based logging framework coupled with phased utility assessment. Evaluated within a sandboxed environment, this approach enables fine-grained analysis of tool-using LLM agents across four dimensions: trajectory propagation, policy compliance, recovery capability, and task completion. By leveraging contamination markers, structured authorization logs, and multidimensional metrics, the framework uncovers hidden containment disparities even when terminal outputs appear identical. Across 17,640 rollback experiments, 73.5% exhibited significant differences in either trajectory or utility. Moreover, the proposed trusted ledger and strong tool-boundary policies elevate authorized task completion rates to 0.8567 and 0.9233, respectively.

Abstract

Tool-using LLM agents process untrusted content, maintain memory, delegate across agents, and invoke side-effecting tools. Existing prompt-injection evaluations typically summarize security with terminal attack or policy outcomes, but equal endpoints can conceal different post-exposure traces and different losses of authorized utility. We introduce ContainmentBench, a sandboxed, trace-based benchmark that separately measures benchmark-defined endpoint policy compliance, instrumented logged propagation, recovery instrumentation, and authorized structured-action completion. In a pre-specified 17,640-rollout study with Qwen2.5-7B-Instruct, all 600 matched active-tainted pairs comparing taint-only and intent-aware enforcement have the same zero committed-harm outcome, yet 73.5% differ in logged trajectory or utility. Taint-only enforcement completes only 0.1642 of authorized tainted workflows; a trusted-ledger policy raises completion to 0.8567, while a strong tool-boundary baseline reaches 0.9233 under the same observed endpoint-policy outcomes. We also find that aggregate logged-spread rankings change with evidence-stage composition and denominator choice. These results show that a terminal policy label is not a sufficient statistic for operational post-exposure containment; evaluations should report endpoint, stage-stratified trajectory, and utility evidence separately, and should promote recovery evidence to comparative claims only where the corresponding controls are valid. The full-scale study is synthetic and single-model; the policy case additionally assumes a correct structured authorization ledger.

Problem

• prompt injection
• post-injection containment
• tool-using LLM agents
• trace-based evaluation
• security benchmark

Innovation

• trace-based evaluation
• post-injection containment
• tool-using LLM agents
• authorized utility completion
• taint propagation