ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?

2026-08-04

AI Summary

It remains unclear whether current large language model (LLM) agents can effectively evolve and improve their skills in continual task settings. This work proposes ContinualSkillBench, a dynamic evaluation framework comprising 500 interrelated, progressively challenging subtasks across five domains, to systematically assess LLMs’ in-context continual skill learning capabilities for the first time. By incorporating in-context continual learning, explicit skill maintenance, cross-task skill reuse analysis, and multi-domain task chaining, the study reveals that sequential task execution generally enhances performance, with in-context learning and explicit skill maintenance yielding comparable results. Moreover, lower-capability models tend to accumulate fragmented skills, whereas explicit skill representations demonstrate clear advantages in tasks requiring skill reuse or high precision.

Abstract

Modern agent frameworks equip large language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five representative domains, each containing 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, in-context learning performs comparably to explicit skill maintenance on average, suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that less capable models tend to accumulate larger, more fragmented collections of task-specific skills. These findings show that current in-context skill evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.

Problem

• continual skill learning
• LLM agents
• skill evolution
• in-context learning
• skill transfer

Innovation

• continual skill learning
• in-context learning
• skill evolution
• LLM agents
• dynamic evaluation framework