Rethinking Continual Experience Internalization for Self-Evolving LLM Agents

2026-06-03

AI Summary

This work addresses the common issue of performance degradation—rather than continual improvement—in large language models during multi-turn experiential learning. To mitigate this, the authors propose a stable and efficient framework for experience internalization, featuring three key innovations: replacing instance-level experiences with principle-level abstractions, employing incremental injection instead of global injection, and introducing an off-policy contextual distillation mechanism. The resulting architecture is both structurally simple and highly robust, significantly enhancing the model’s capacity for sustained learning and tool utilization across iterative interactions. This approach thus enables large language models to achieve sustainable self-evolution through repeated engagement.

Abstract

Experience internalization converts contextual experience from past interactions into reusable parametric capability, offering a promising path toward continual learning in large language models (LLMs). While prior work has predominantly focused on single-iteration transfer, we discover that under multi-iteration experience learning, existing methods suffer from a progressive capability collapse rather than compounding improvement. We systematically examine this failure through three vital dimensions of experience internalization: (1) Experience Granularity: We find that principle-level experience is more durable than instance-level experience, as it effectively abstracts transferable strategies away from trajectory-specific details. (2) Experience Injection Pattern: Our analysis reveals that step-wise injection significantly outperforms global injection by aligning experience with intermediate decision states, a property that is critical for long-horizon tool use. (3) Internalization Regime: We demonstrate that off-policy context-distillation on high-quality teacher trajectories provides a substantially more stable training signal than on-policy context-distillation, which is inherently limited by local corrections on student-induced flawed states. Together, these insights yield a simple yet robust recipe for stable and sustainable experience internalization, providing concrete guidance for engineering self-evolving and continually learning LLMs.

Problem

• continual learning
• experience internalization
• large language models
• capability collapse
• self-evolving agents

Innovation

• experience internalization
• continual learning
• step-wise injection
• off-policy distillation
• principle-level abstraction