Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents

2026-08-03

AI Summary

This work proposes EvoBreak, a novel attack method that exploits the emergent security risks in self-evolving large language model agents. Although individual experiences acquired during benign interactions appear harmless, their strategic combination can inadvertently erode safety boundaries and create covert attack surfaces. EvoBreak orchestrates a sequence of seemingly innocuous tasks to guide the agent toward accumulating complementary experiences, which are then jointly activated in a final query to bypass safety constraints. This approach establishes a highly stealthy, serialized attack paradigm that requires no direct memory access and leverages only benign-looking interactions. Extensive experiments demonstrate that EvoBreak consistently outperforms existing attacks across diverse self-evolution frameworks, model architectures, pre-evolution domains, and safety benchmarks, while maintaining high benignity throughout the attack process.

Abstract

Self-evolving large language model agents improve their capabilities by distilling interaction trajectories into persistent experiences. Yet this mechanism introduces a new safety risk: experiences that are benign in isolation may jointly weaken an agent's safety boundary when accumulated and reused across sessions. Existing memory attacks typically require direct memory access or induce explicitly malicious records, limiting their stealthiness and applicability. We propose EvoBreak, an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences. EvoBreak repeatedly observes the experiences distilled by the victim, identifies uncovered target-relevant requirements, and adaptively acquires complementary experiences before reformulating the final query to activate them jointly. To support training, we introduce BreakGym, a structure-first synthesis pipeline that generates decomposable safety-sensitive targets with diverse dependency structures. EvoBreak is optimized using rejection-sampling supervised fine-tuning and Hint-guided GRPO. Experiments across self-evolving frameworks, victim backbones, pre-evolution domains, and safety benchmarks demonstrate that EvoBreak consistently outperforms existing attacks while maintaining high benignness. These results reveal benign experience composition as a persistent attack surface in self-evolving agents.

Problem

• self-evolving agents
• experience composition
• safety boundary
• memory attacks
• benign experiences

Innovation

• experience composition
• self-evolving agents
• stealthy attack
• safety boundary
• sequential adversarial strategy