SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

2026-08-05

AI Summary

Existing evaluation benchmarks struggle to effectively validate the full execution process of skill-augmented agents in long-horizon tasks, lacking an integrated assessment framework that jointly considers task-time skills, verifiable artifacts, and environmental context. To address this gap, this work introduces SkillTV-Bench, a benchmark comprising 681 real agent trajectories across 50 tasks spanning 11 domains, along with SkillTV-Evolve—a novel method that, for the first time, integrates task-time skills with verifiable trajectories. SkillTV-Evolve employs a reusable and evolvable JudgeSkill mechanism to guide evaluators toward targeted, evidence-based trajectory validation. Experimental results demonstrate that the refined JudgeSkill improves evaluator accuracy by 14.8 percentage points and increases the success rate of offline trajectory selection from 22.9% to 45.5% over ten replay attempts.

Abstract

LLM agents increasingly execute long-horizon tasks through tool use and environment interaction, shifting evaluation from final-response scoring to verification of complete executions. For skill-augmented agents, verification additionally requires the procedural knowledge encoded in task-time skills, because this knowledge indicates what evidence to inspect and which failures are task-critical. However, existing judge benchmarks often expose final responses or static trajectories, and rarely combine task-time skills with directly inspectable artifacts and environments. We therefore introduce SkillTV-Bench, a 681-case benchmark of real agent trajectories from 50 tasks across eleven domains, designed to evaluate skill-aware trajectory verification for both LLM-as-a-Judge and Agent-as-a-Judge methods. Additionally, we propose SkillTV-Evolve, which externalizes verification knowledge as a reusable JudgeSkill that guides an agent judge to plan targeted inspections and issue evidence-grounded verdicts. On a disjoint development pool, an automated evolution loop further refines the JudgeSkill using misjudged cases. On SkillTV-Bench, the refined skill increases the same agent judge's accuracy by 14.8 percentage points. In offline rollout-pool selection, it increases selected-trajectory success from 22.9% with one rollout to 45.5% with ten rollouts. The code and data are available at https://github.com/HanZhi306/SkillTV-Bench

Problem

• skill-augmented agents
• trajectory verification
• judge benchmarks
• procedural knowledge
• execution evaluation

Innovation

• skill-augmented agents
• trajectory verification
• JudgeSkill
• SkillTV-Bench
• SkillTV-Evolve