SkillEval: Decomposing Agent Skill Quality into Interpretable Signals

2026-08-07

AI Summary

This work addresses the limitations of existing skill evaluation methods, which rely on downstream task performance and fail to provide a comprehensive, interpretable assessment of intrinsic skill quality. To overcome this, the authors propose SkillEval, a novel framework that decouples skill quality into multiple interpretable semantic attributes. SkillEval learns fixed scoring directions in the model’s latent space by contrasting pairs of skills, enabling fine-grained attribute-level evaluation through a projection mechanism. Crucially, it controls for confounding factors such as document length and formatting to minimize interference. Experimental results demonstrate that SkillEval reliably discriminates between skill qualities, with its scores strongly correlating with downstream task performance. Moreover, the framework effectively guides skill refinement, leading to significant improvements in task success rates.

Abstract

Agent skills provide reusable procedural knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality by testing whether a skill improves performance on specific downstream tasks. However, a reusable skill may apply to multiple task scenarios. Downstream evaluation mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspect of the skill should be improved. We find that general properties of the \texttt{SKILL.md} document play an important role in skill quality. To evaluate these properties, we propose \textbf{SkillEval}, an interpretable framework for document-level skill evaluation. SkillEval evaluates each property using a fixed and inspectable scoring direction, producing interpretable scores. It further measures and reduces the influence of unrelated document features, such as length and formatting, so that each score captures its intended semantic property more specifically. Specifically, SkillEval learns an interpretable direction for each quality property from controlled positive--negative skill pairs in the hidden representation space of the model, and scores a new skill by projecting its representation onto these fixed directions. We use SkillEval to evaluate skills in controlled quality tests and show that SkillEval reliably distinguishes skills of different quality. In addition, SkillEval scores closely reflect downstream task performance, providing an early indication of whether a skill is likely to help an agent complete a task. We further explore SkillEval for diagnosing weaknesses in skill documents and guiding targeted revisions. The revised skills improve the targeted properties and achieve higher pass rates on downstream tasks.

Problem

• skill evaluation
• interpretable assessment
• agent skills
• document-level quality
• downstream task compatibility

Innovation

• SkillEval
• interpretable evaluation
• skill quality
• document-level assessment
• representation projection