How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

2026-08-07

AI Summary

This work addresses the challenge in multi-turn agent reinforcement learning of simultaneously assigning credit at both the trajectory-level action and the intra-action token levels. The authors propose FACTOR, a novel method that decouples these two credit assignment problems: it estimates action-level credit using calibrated temporal difference (TD) residuals and allocates token-level credit through a feedback-conditioned teacher–student likelihood discrepancy. To mitigate token-length bias and preserve credit sign consistency, FACTOR incorporates action-mean reduction and a credit conservation mechanism. Notably, the approach requires no hyperparameter tuning and demonstrates robust generalization across diverse environments, backbone architectures, and model families. Empirical results show consistent and significant improvements over strong baselines on ALFWorld, WebShop, and ScienceWorld benchmarks, with particularly pronounced gains in long-horizon tasks.

Abstract

Credit assignment in multi-turn agent reinforcement learning operates at two levels: assigning trajectory-level credit to actions and distributing each action's credit across its tokens. In this paper, we introduce FACTOR, which separates these decisions. FACTOR uses checkpoint-calibrated TD residuals to assign per-action credits that telescope to the trajectory advantage, and feedback-conditioned teacher-student likelihood gaps to allocate each credit across the realized action tokens. Per-action normalization preserves the action-average coefficient and prevents token-level sign flips. We pair this construction with an action-mean reduction, removing the implicit dependence of an action's scalar surrogate weight on its token length. At the behavior policy and before clipping, each action's inner action-mean surrogate equals its TD credit. FACTOR consistently improves over competitive baselines across ALFWorld, WebShop, and ScienceWorld, with every environment-seed comparison favoring FACTOR and the largest gains emerging on the longest-horizon environment. The same hyperparameters transfer without retuning to a larger backbone and to a different model family. Ablations identify TD action credit as the dominant driver of the improvement, with hindsight token allocation contributing complementary gains.

Problem

• credit assignment
• multi-turn reinforcement learning
• action-to-token allocation
• trajectory-level credit
• token-level credit

Innovation

• credit assignment
• multi-turn reinforcement learning
• token-level allocation
• TD residuals
• teacher-student likelihood gap