Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

2026-09-30

AI Summary

This study addresses the underutilization of failure trajectory information and the difficulty in precisely localizing and correcting erroneous decisions in LLM agents. To this end, it pioneers a large-scale cross-environment error diagnosis paradigm for agents. Specifically, an AET pipeline is introduced to automatically collect multi-environment interaction logs and generate diagnosis-correction plans, yielding the AED dataset comprising 50,000 samples. Furthermore, this work proposes an execution-evidence-based automated diagnostic verification mechanism alongside a decoupled diagnosis-repair training strategy, optimizing models through replay validation and supervised fine-tuning. Experimental results demonstrate that the proposed correction approach improves verifier pass rates by 32.7 percentage points, while Qwen3-8B achieves a diagnostic accuracy of 63.6%, significantly outperforming baselines.

Abstract

An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.

Problem

• LLM agent failure
• error diagnosis
• failure analysis
• error-aware post-training
• agent error dataset

Innovation

• Agent Error Dataset
• Error-Diagnosis Pairs
• Agentic Error-to-Training Pipeline
• Failure Analysis
• Error-Aware Post-Training