Exploring Expert Failures Improves LLM Agent Tuning

2025-04-17

AI Summary

To address the persistent out-of-distribution (OOD) challenges and performance bottlenecks of LLM-based agents in complex subtasks, this paper proposes leveraging *informative planning patterns and critical actions* extracted from expert (e.g., GPT-4) *failure trajectories*, rather than relying solely on successful ones. Methodologically, we introduce the first systematic failure trajectory analysis framework: (i) action-level failure attribution to identify beneficial decisions, (ii) a harmful-action filtering mechanism, and (iii) integration of high-quality filtered failure segments with successful trajectories into a rejection sampling fine-tuning (RFT) pipeline. This paradigm substantially enhances exploration efficiency and acquisition of pivotal skills. Experiments demonstrate state-of-the-art performance: 62.0% success rate on WebShop (+8.4% over RFT, +26.4% over GPT-4), the first agent to exceed 0.81 normalized score on WebShop and 81 points on SciWorld—establishing new SOTAs on both benchmarks.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) have shown tremendous potential as agents, excelling at tasks that require multiple rounds of reasoning and interactions. Rejection Sampling Fine-Tuning (RFT) has emerged as an effective method for finetuning LLMs as agents: it first imitates expert-generated successful trajectories and further improves agentic skills through iterative fine-tuning on successful, self-generated trajectories. However, since the expert (e.g., GPT-4) succeeds primarily on simpler subtasks and RFT inherently favors simpler scenarios, many complex subtasks remain unsolved and persistently out-of-distribution (OOD). Upon investigating these challenging subtasks, we discovered that previously failed expert trajectories can often provide valuable guidance, e.g., plans and key actions, that can significantly improve agent exploration efficiency and acquisition of critical skills. Motivated by these observations, we propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories and integrates them into the training dataset. Potentially harmful actions are meticulously excluded to prevent contamination of the model learning process. By leveraging the beneficial actions in expert failures, EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance. Remarkably, our approach achieved a 62% win rate in WebShop, outperforming RFT (53. 6%) and GPT-4 (35. 6%), and to the best of our knowledge, setting a new state-of-the-art as the first method to surpass a score of 0.81 in WebShop and exceed 81 in SciWorld.

Problem

• Enhances LLM agent tuning using expert failure insights
• Addresses persistent OOD challenges in complex subtasks
• Improves agent performance by filtering harmful expert actions

Innovation

• Leverages failed expert trajectories for guidance
• Excludes harmful actions to prevent contamination
• Integrates beneficial actions into training dataset