Data Leakage Prevention in Agentic Applications via Preemptive Hardening

2026-07-21

AI Summary

This work addresses critical security vulnerabilities in multi-agent systems arising from prompt injection attacks and failures at instruction/data boundaries, which can lead to data leakage and tool misuse—particularly challenging to mitigate in heterogeneous agent workflows spanning diverse codebases. The paper introduces the first automated pre-deployment defense framework tailored for multi-agent applications. By statically analyzing prompt templates, tool interfaces, and invocation code, the framework identifies high-risk leakage patterns and synthesizes minimally invasive patches, including boundary sanitization, allowlist-based gating, and least-privilege checks. Validated against both adversarial and benign inputs, the approach ensures functional integrity without runtime overhead. Empirical evaluation on five real-world applications and the AgentDojo benchmark demonstrates complete prevention of data leakage under basic attacks and a 91% reduction under stress-induced attacks, all while preserving original system functionality.

Abstract

Agentic systems integrate LLM driven planning with interfaces to external tools, making data leakage and tool misuse feasible via instruction/data boundary failures and prompt injection attacks. Enforcing required controls consistently is particularly challenging in workflows spanning many codebases and heterogeneous agents. To address this challenge in multi agentic systems, we present a pre-deployment pipeline for scanning, hardening, and validation of agentic applications. The pipeline analyzes prompt templates, tool interfaces, and tool-invocation code to identify leakage-enabling patterns and generate actionable patches. The hardened application is then validated through adversarial prompt injection attacks and benign input variations ensuring that mitigations do not disrupt intended behavior. In the hardening stage, high-risk tools are prioritized, and minimally invasive mitigations are applied, including schema tightening, boundary sanitization, allowlist-based tool gating, and least-privilege checks. In the validation stage, the pipeline automatically generates attack inputs that mimic jailbreaks, instruction overrides, and tool-targeted manipulation, along with benign task variants, to confirm that the functionality of the hardened application is preserved after remediation. We evaluated the pipeline on five real-world agentic applications, as well as on the AgentDojo benchmark. Across all applications, the proposed pipeline identified recurring leakage-enabling patterns and generated patches that can be integrated without disrupting the intended application behavior. The resulting modifications of application code were shown to eliminate leaks when targeted by basic jailbreak and instruction-override attacks, achieving a 100% reduction in leakage, and reduce leaks by 91% under conditions of stress-induced manipulation, without the need of continuous runtime policy enforcement.

Problem

• data leakage
• agentic systems
• prompt injection
• tool misuse
• security hardening

Innovation

• preemptive hardening
• data leakage prevention
• agentic systems
• prompt injection defense
• tool misuse mitigation