ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

2026-07-30

AI Summary

This work addresses the limitations of existing evaluation methods that focus solely on final outcomes, which fail to distinguish reliable reasoning from accidental success or diagnose process-level flaws in long-horizon tasks. To this end, we propose ClawTrack, a dual-dimensional evaluation framework that jointly assesses task completion (Task Score) and reasoning process quality (Process Score). Spanning 320 tasks across eight domains, ClawTrack introduces fine-grained, stepwise scoring along four dimensions, enabling the first interpretable, process-level evaluation of autonomous agent reasoning trajectories. Our Process Grader combines rule-based logic with large language models, incorporating 12,541 task-specific scoring criteria and integrating over 25 deterministic simulation environments. Validation across 21 models and more than 16,000 trials demonstrates that process scores effectively attribute success or failure, filter out spurious successes, and—when used to select high-quality reasoning trajectories—significantly boost performance across model scales, with consistent results across different evaluator models.

Abstract

As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.

Problem

• autonomous agents
• trace-level evaluation
• process attribution
• reasoning reliability
• long-horizon tasks

Innovation

• trace-level evaluation
• process scoring
• autonomous agents
• reasoning attribution
• trajectory filtering