Progressive Agent Skill Generation via Reinforcement Learning

2026-08-03

AI Summary

Existing learning-based agents lack direct supervision signals for skill acquisition, rendering skill quality assessable only indirectly through downstream task performance. This work proposes a progressive skill generation framework that models skill construction as an editable sequential process. It leverages reinforcement learning to optimize each editing operation and introduces a rollback reward mechanism: by comparing the downstream execution outcomes of a skill before and after editing on an anchor query, the framework dynamically decides whether to retain the modification. This approach eliminates the need for heuristic rules tailored to different evidence sources, enabling a unified and directly evaluable skill generation pipeline. Experiments demonstrate that the method outperforms the strongest baseline by 3.3 and 6.7 percentage points in average downstream task success rates on CL-Bench and Tau2-Bench, respectively, significantly surpassing existing heuristic and pipeline-based approaches.

Abstract

Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-$α$, a reinforcement learning method for progressively generating high-quality agent skills. Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits, and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill-$α$ generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill-$α$ improves average downstream success rates over the strongest skill-generation baseline by 3.3 points on CL-Bench and 6.7 points on tau2-bench. Further ablations validate the importance of rollback reward and progressive generation.

Problem

• skill generation
• reinforcement learning
• supervision signal
• downstream tasks
• agent skills

Innovation

• progressive skill generation
• reinforcement learning
• rollback reward
• agent skills
• sequential editing