Fantastic Adaptive Taxonomies and How to Use Them

2026-07-17

AI Summary

This work addresses the challenge of effectively reusing failure feedback from existing agent execution trajectories, which are often lengthy, instance-specific, and lack standardized failure descriptions. To overcome this, the authors propose an unsupervised method that automatically distills raw trajectories into a structured, evidence-backed failure taxonomy. This taxonomy forms an adaptive failure glossary organized along three axes—system-level, role-level, and domain-level—and serves as a unified feedback interface integrated into trajectory selection, runtime monitoring, and system search processes. Requiring no manual annotation, the glossary achieves a 10× compression ratio while exhibiting semantics closely aligned with expert annotations. Empirical results demonstrate significant performance gains across multiple benchmarks: SWE-agent’s resolution rate improves from 60% to 70%, Claude Code reaches 70.7%, and Terminal-Bench 2.0 accuracy increases by 8–15 points.

Abstract

An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback. Yet raw traces are a poor medium for accumulating that feedback: long, instance-specific, and lacking a stable vocabulary for recurring failures. We argue that an agent system should instead maintain an explicit representation of how it fails, induced from its own behavior and reusable wherever failure feedback is needed. AdaMAST builds this representation by converting a target system's traces into a compact, evidence-grounded failure taxonomy: named failure codes organized along three fixed axes (system-level, role-specific, and domain-specific), with every name, definition, and evidence pattern induced from the traces; no code is hand-authored, no trace human-annotated. The taxonomy is not merely a post-hoc diagnostic but a shared feedback interface, improving agents in three ways. In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form reflection on all five benchmarks we test. At runtime, taxonomy feedback raises SWE-agent's resolution on SWE-bench Verified Mini from 60% with free-text reflection to 70%, and improves Claude Code from 64.0% to 70.7% as a runtime skill. In trajectory selection, AdaMAST-Judge, a verifier built on the induced codes, improves best-of-5 accuracy on Terminal-Bench 2.0 by 8-15 points over Pass@1. The vocabulary itself is compact (an order-of-magnitude compression that preserves trace distinctions), human-faithful (matching expert failure annotations more closely than a hand-crafted reference vocabulary), and adaptive (taxonomies induced for different domains share few codes). Adaptive failure taxonomies close the loop between the traces agents produce and the procedures that improve them.

Problem

• execution traces
• failure feedback
• adaptive taxonomies
• agent systems
• failure representation

Innovation

• adaptive failure taxonomy
• execution trace analysis
• agent system improvement
• evidence-grounded categorization
• runtime feedback