Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

2026-09-21

AI Summary

研究提出Critical-State RL方法，通过识别多轮交互中的可训练状态并优化策略，有效解决了多轮工具使用中因单次模型调用导致的失败问题。

Abstract

Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions. Given task-defined candidate calls and local rewards, the method assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible. It then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states using contextual-bandit training. Experiments on the Berkeley Function Calling Leaderboard (BFCL) v4 compare training at diagnostic-selected states with training at alternative states. For missing-function tasks, the diagnostic selects the response after the tool becomes available; for missing-argument tasks, it selects the response before the missing argument is supplied. Training the selected responses improves performance, including about 14 percentage points on the missing-function task, while training the alternatives leaves performance flat or worse. We further apply the recipe across models and tasks, including logged repeat-call avoidance and memory management.

Problem

• multi-turn tool-use
• reward variation
• trainable states

Innovation

• Critical-State RL
• multi-turn interactions
• contextual-bandit training
• reward variation