SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

2026-08-03

AI Summary

Current language model agents struggle to effectively recognize, invoke, and coordinate multiple skills to accomplish complex tasks. This work proposes the first scalable, verification-based synthetic data generation framework that automatically constructs 4,000 skill-oriented tasks and 27,164 high-quality executable trajectories from 2,000 publicly available skills. The framework integrates rule-based validation, agent self-feedback repair, supervised fine-tuning, and cross-agent interface evaluation to ensure data fidelity and task complexity. The resulting SkillEval benchmark supports both single-skill and multi-skill compositional tasks, significantly enhancing diverse models’ skill invocation capabilities across multiple scenarios. Moreover, performance consistently improves as the breadth of covered skills expands, demonstrating the framework’s effectiveness in fostering scalable, skill-aware agent behavior.

Abstract

Agent skills have become an important mechanism for equipping language-model agents with reusable procedural knowledge. However, providing skills alone does not guarantee that current models can effectively identify, apply, and coordinate them. To improve skill-use capabilities, we introduce SKT, a verified data synthesis pipeline that constructs skill-grounded tasks and executable trajectories from large collections of agent skills. SKT selects suitable single-skill and multi-skill configurations, synthesizes tasks through rule-based and agent-based verification with feedback-guided repair, and retains only successful trajectories that substantially use every required skill. Using 2,000 public skills, SKT produces 4,000 task packages and 27,164 verified trajectories. Based on the same pipeline and a disjoint test pool, we further construct SkillEval, a held-out executable benchmark for evaluating skill use. Experiments across diverse models, benchmarks, and agent harnesses show that supervised fine-tuning on SKT-generated trajectories consistently improves skill-use performance. Verification ablations, cross-harness evaluation, and scaling experiments further demonstrate that these gains depend on high-quality supervision, extend beyond a single agent interface, and increase with broader skill coverage. Together, these results establish verified data synthesis as an effective and scalable approach for skill-use training.

Problem

• skill-use
• language-model agents
• procedural knowledge
• agent capabilities
• skill coordination

Innovation

• verified synthetic data
• skill-use training
• executable trajectories
• agent skills
• data synthesis pipeline