SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents

2026-06-11

AI Summary

This work addresses the mismatch between fixed task distributions and evolving agent policies during tool-augmented reinforcement learning, which often leads to inefficient interactions. To overcome this limitation, the authors propose SENTINEL, a novel framework that establishes a Controller–Proposer–Solver closed-loop mechanism to systematically transform agent failure trajectories into high-information, executable training tasks, thereby enabling failure-driven adaptive reinforcement learning. By integrating trajectory analysis, error pattern abstraction, and targeted task generation, SENTINEL dynamically focuses training on the model’s weaknesses to enhance generalization. Evaluated on Tau2-Bench Retail using Qwen3-4B-Thinking-2507, the approach improves Pass⁁1 from 66.4% to 74.9% and significantly outperforms conventional reinforcement learning methods on general synthetic tasks.

Abstract

Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution. When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks. SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass\^{}1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Pass\^{}k metrics. These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.

Problem

• tool-using language model agents
• reinforcement learning
• task distribution mismatch
• failure-driven training
• rollout inefficiency

Innovation

• failure-driven reinforcement learning
• tool-using language agents
• targeted task generation
• trajectory analysis
• Controller-Proposer-Solver framework