Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember

2026-07-31

AI Summary

Traditional self-play agents lack persistent memory and struggle to learn continuously from failures, while existing skill-memory approaches rely on fixed task distributions and cannot evolve dynamically. This work proposes the SESA framework, which for the first time jointly evolves skill memory and self-play task generation: a challenger dynamically proposes tasks, and a solver retrieves and distills past failure experiences into reusable skills stored in a memory bank, enabling bidirectional co-optimization of policy training and future task distribution. SESA supports both memory-free deployment and on-demand retrieval. Evaluated on seven open-domain multi-hop question answering benchmarks, SESA outperforms SSP by 1.2–3.2 points and SkillRL by 0.9 points on average. Notably, even without the memory bank (SESA-Off), it retains gains of 1.8–2.2 points, demonstrating that skill evolution fundamentally enhances policy learning.

Abstract

Self-play agents can generate training problems without questions from target benchmarks, but their curricula lack persistent state: failures affect gradients yet do not explicitly shape future practice. External skill memories preserve procedural experience but are typically learned from fixed task distributions. We introduce \textbf{SESA} (Self-Evolving Skill-Augmented Agent), which makes procedural memory an evolving state of tool-augmented search self-play. A challenger poses problems, while a separately parameterized solver alone retrieves skills. Informative failures are distilled into reusable skills and written back to memory. The updated memory changes solver behavior and success, which changes the challenger's reward and the distribution of future problems; the resulting frontier produces new failures that rewrite memory. This bidirectional loop makes task generation and skill memory co-evolve. Because retrieved skills shape on-policy training trajectories, their benefits can enter the model parameters as well as remain in the external bank, enabling memory-free deployment and optional inference-time retrieval. Across seven open-domain and multi-hop question-answering benchmarks, SESA improves average accuracy over SSP by 1.2--3.2 points across multiple backbones and surpasses the skill-augmented SkillRL baseline by 0.9 points under a unified evaluation protocol. On Qwen3 models, SESA-Off retains 1.8--2.2 points of improvement over SSP, while the final skill bank adds a further 0.5--1.0 points. These results show that evolving skill memory is not merely an inference-time plug-in: it changes policy learning and the future training distribution while retaining value as optional external memory. Our code is available at https://github.com/Zenghuang-Fu/SESA-Self-Evolving-Search-Agents.

Problem

• self-play
• skill memory
• task generation
• procedural memory
• training distribution

Innovation

• self-play
• skill memory
• co-evolution
• tool-augmented search
• procedural memory