SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

2026-07-16

AI Summary

This work addresses the fragmentation, redundancy, and inconsistent quality of open-source LLM agent skills documented in community-maintained SKILL.md files, which lack systematic integration and empirical validation. To remedy this, the authors propose SkillCorpus, a multi-stage pipeline that aggregates, filters, and structures 821,000 community-contributed skills into a high-quality corpus of 96,401 curated skills. The framework introduces a retrieval-and-selection mechanism grounded in a 16-category taxonomy and evaluated along three dimensions: utility, robustness, and safety. For the first time, it enables end-to-end validation of the effectiveness boundaries of community skills in real-world agent tasks. Evaluated on SkillsBench, GDPVal, and QwenClawBench, SkillCorpus consistently improves performance across all benchmarks, achieving an absolute gain of 7.5 percentage points on SkillsBench.

Abstract

Agent skills, SKILL.md files that package reusable procedural knowledge for an LLM agent, are a popular mechanism for extending agent capabilities. Public repositories now host them in large and growing numbers, yet these artifacts are fragmented, redundant, and uneven in quality, and their value in practice is unclear. A core question remains open, namely how to consolidate this open-source SKILL.md ecosystem into a single usable corpus, and what bounds its benefit on real-world agent tasks. We present SkillCorpus, a framework that aggregates, curates, matches, and evaluates the open skill ecosystem at scale. It filters ~821,000 crawled skills through a multi-stage pipeline into 96,401 skills organised by a 16-class taxonomy and three quality facets (utility, robustness, safety), and pairs them with a fine-tuned retrieval-and-selection stack that matches task-relevant skills. We evaluate end-to-end across three benchmarks (SkillsBench, GDPVal, QwenClawBench), two harnesses, and two open backbones with a frontier robustness check. Integrating SkillCorpus yields consistent gains across all three benchmarks, largest on SkillsBench (+7.5 pp). An operational analysis traces the gains to a coverage boundary and a harness boundary. SkillCorpus is, to our knowledge, the first end-to-end account of when a curated, retrieval-served community corpus improves real agent tasks, and where it does not. The dataset, models, and code will be released upon acceptance.

Problem

• LLM agents
• skill ecosystem
• skill consolidation
• open-source skills
• agent task performance

Innovation

• SkillCorpus
• LLM agents
• skill retrieval
• open skill ecosystem
• agent benchmarking