Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

2026-08-05

AI Summary

Current evaluations lack fine-grained metrics for assessing large language models’ (LLMs’) autonomous recognition and correct usage of skills. This work proposes the Skill-Use (SU) benchmark, which, for the first time, decouples skill utilization into three quantifiable dimensions: triggering, compliance, and boundary adherence, and introduces an integrated SU scoring mechanism. Leveraging a Docker sandbox environment, real-world skill–task pairings, and a multi-agent framework, we evaluate eight LLMs across 79 skills and 177 tasks. Our results reveal that even the best-performing configuration achieves only an SU score of 0.613, highlighting both triggering and compliance as distinct bottlenecks. Moreover, performance is found to be highly dependent on the agent framework rather than the underlying model itself.

Abstract

Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.

Problem

• skill use
• large language models
• agent evaluation
• procedural compliance
• trigger recognition

Innovation

• Skill-Use
• LLM agents
• progressive disclosure
• procedural compliance
• agent harnesses