ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems

2026-10-08

AI Summary

This study addresses the challenges of tracing early-step errors in LLM agents and the limited attribution capability of existing hidden-state representations by proposing ReCast. This method extracts complementary pattern and bias features through layer selection and feature engineering, and trains an encoder via contrastive learning with a ranking objective to transform frozen LLM hidden states into step-level representations tailored for root cause localization. The main contributions include the ReCast framework and the release of the ReCast-2K dataset. Experimental results demonstrate that the proposed approach achieves state-of-the-art performance on the Hit@1 metric across four benchmarks, outperforming the strongest baseline by 5.65 and 9.19 percentage points, respectively.

Abstract

In LLM-based agent systems, failures can originate from early steps whose effects propagate through subsequent interactions, making their origins difficult to identify. To trace such failures back to their origin, failure attribution has been formulated as the task of identifying the earliest step responsible for the failure. Recent methods leverage LLM internal signals for failure attribution, typically using hidden states as step representations. We therefore conduct an empirical study to evaluate how effectively these representations distinguish root-cause steps from other steps and find limited separation. Motivated by this observation, we propose ReCast, a step representation learning method that transforms hidden states from a frozen LLM into attribution-oriented step representations. ReCast first selects attribution-relevant layers, then constructs complementary pattern and deviation features, and finally learns contextualized step representations through an encoder trained with contrastive and ranking objectives. We also introduce ReCast-2K, a training dataset for failure attribution. ReCast achieves the best Hit@1 across four benchmarks, surpassing the strongest baseline by 5.65 and 9.19 pp on Who&When Algorithm and Handcrafted, respectively. Code is available at https://anonymous.4open.science/r/ReCast-5FB6 .

Problem

• failure attribution
• LLM-based agent systems
• step representation learning
• root-cause identification

Innovation

• Failure Attribution
• Step Representation Learning
• Contrastive Learning
• LLM-based Agent Systems
• Ranking Objectives