🤖 AI Summary
This work addresses the lack of systematic evaluation of agent skills' practical utility in cross-domain, multi-model settings. We propose the first scalable evaluation framework specifically designed for agent skills, enabling skill developers to define custom tasks and evaluation dimensions grounded in real-world scenarios. The framework quantifies the enhancement provided by skills to large language model (LLM) agents through metrics of instruction following and task completion. It integrates automated task generation, scoring rules, and comparative experiments across 19 commercial and open-source LLMs, yielding a benchmark comprising 1,000 diverse tasks. Experimental results reveal significant disparities among models in adhering to skill instructions, thereby validating the effectiveness of skills in guiding agent behavior. The benchmark dataset is publicly released to foster further research in this area.
📝 Abstract
Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic tasks to rigorously assess the aspects of a skill that matter most to them, and that estimates skill utility by solving those tasks. Further, we apply our evaluation approach at scale to 500 real-world skills, generating 1,000 tasks derived from the skills' content, along with instruction-following and goal-completion scoring rubrics. Using these metrics, we evaluate how 19 agent-model configurations, both proprietary and open-source, perform on the tasks. Our results show that models vary widely in how closely they adhere to the instructions encoded in skills, leading to substantial differences in their performance gains. Furthermore, we show that access to a skill significantly changes model behavior compared to the no-skill setup, providing an essential mechanism for encoding opinionated workflows into LLM agents. We release our evaluation dataset to support future work on agent skills.