ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

2026-05-31

AI Summary

Current reinforcement learning agents lack reusable, cross-task generalizable skills, and the decoupling between skill acquisition and policy optimization often leads to conflicting objectives. This work proposes ReSkill, a framework that establishes, for the first time, a closed-loop synergy between skill generation and policy learning. By embedding an assertion-driven skill generator within policy optimization, ReSkill automatically creates condition-triggered skills through failure diagnosis. It further employs intra-group rollback sampling and adaptive-discount Thompson sampling to enable automatic skill evaluation, refinement, and pruning. The group-structured design based on GRPO ensures that skills continuously support policy evolution. Empirical results demonstrate that ReSkill significantly outperforms existing methods across multiple domains, with the largest performance gains observed on unseen tasks, thereby validating the efficacy of co-evolving skills and policies.

Abstract

Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.

Problem

• agentic reinforcement learning
• skill creation
• policy optimization
• reusable strategies
• skill-policy co-evolution

Innovation

• skill-policy co-evolution
• assertion-driven skill creation
• within-group rollout sampling
• Thompson Sampling with adaptive discounting
• modular skills