HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

2026-08-07

AI Summary

This work addresses latent security risks introduced by persistent components—such as memory and tools—in agent frameworks, where malicious payloads can remain dormant across tasks and trigger harm in subsequent benign requests. The study presents the first systematic modeling of such cross-component delayed risks, introducing the concept of a “persistence risk lifecycle” and constructing a benchmark comprising 328 executable cases spanning seven carrier types. Employing a multi-stage evaluation method based on execution traces, the framework tracks attacks from initial injection through persistence to eventual policy violation. Experimental results reveal that defensive efficacy is highly dependent on both carrier type and agent–model configuration, and that relying solely on end-to-end attack success rates obscures critical differences in risk propagation dynamics, thereby offering fine-grained insights for secure agent system design.

Abstract

Modern agent harnesses persist state across tasks and sessions through persistent carriers like memory, skills, tools, and shared artifacts. However, this capability creates delayed safety risks: attacker-influenced content can cross system boundaries and later affect the execution of a benign request. Existing benchmarks typically focus on a few carriers or harnesses, while end-to-end attack-success rates reveal little about how risks propagate. To this end, we present HarnessSafe, a benchmark comprising 328 executable cases across seven persistent-carrier families and evaluated on most mainstream agent harnesses. Each case is specified as a Persistent-Risk Lifecycle that traces attacker influence from its initial entry, through persistence across carriers and system boundaries, to a later benign trigger and an observable violation. We further introduce a multi-stage, trace-based evaluation that uses observable execution evidence to determine how far each attack chain progresses and where it is stopped. Experiments show that containment is carrier-specific and strongly depends on the harness-model configuration. Both the harness and model backend substantially shape containment outcomes, while attack success rates cannot reflect distinct lifecycle progression patterns.

Problem

• persistent carriers
• agent harnesses
• safety risks
• attack propagation
• delayed security

Innovation

• persistent carriers
• agent safety
• risk lifecycle
• trace-based evaluation
• HarnessSafe