Rethinking Self-Evolution: A Constrained Exploration-Exploitation Process for Mitigating Skill Overfitting

2026-07-29

AI Summary

This work addresses the challenges faced by large language model (LLM) agents when reusing past experiences in real-world environments—namely, overfitting due to sparse trajectories and performance degradation on previously mastered tasks caused by unconstrained exploration. To mitigate these issues, the authors propose SkillBoost, a three-stage framework that formulates skill self-evolution as a constrained exploration–exploitation process for the first time. The approach begins with structured analysis to identify skill deficiencies, leverages LLM priors to generate diverse repair candidates, and only accepts updates validated within a regression boundary to preserve prior capabilities. Evaluated across 23 model–benchmark configurations, SkillBoost significantly outperforms both handcrafted and LLM-generated skills, while producing optimized skills that exhibit strong cross-agent transferability.

Abstract

Enabling large language model (LLM) agents to accumulate and reuse experience from past interactions remains a central challenge in real-world applications. A promising solution is to treat skills as trainable states and optimize them in the same way as model parameters in neural network training. However, data-driven skill optimization is prone to overfitting to the limited trajectories collected from real environments. Overexploiting these trajectories overfits the current batch, while unconstrained exploration causes regression on previously solved cases. This tension motivates a constrained search view of skill self-evolution, governed by an exploration--exploitation trade-off. We propose SkillBoost, a three-stage framework that mitigates both risks: structured exploitation localizes observed failures to editable skill components, prior-guided exploration draws on prior knowledge in the LLM to generate diverse repair candidates, and verified acceptance commits a candidate only when it improves performance within a regression bound. Experiments across 23 model--benchmark configurations show that SkillBoost achieves state-of-the-art performance while mitigating overfitting, outperforming both human-crafted and LLM-generated skills. Transfer experiments further show that optimized skills can be reused by other agents on similar tasks.

Problem

• skill overfitting
• exploration-exploitation trade-off
• LLM agents
• experience reuse
• trajectory optimization

Innovation

• skill self-evolution
• exploration-exploitation trade-off
• overfitting mitigation
• SkillBoost
• transferable skills