temporal credit assignment

To address the granularity imbalance in credit assignment for large language model reinforcement learning—where token-level methods (e.g., PPO) suffer from inaccurate advantage estimation due to critic training instability, and trajectory-level methods (e.g., GRPO) lack precision by relying solely on terminal rewards—this paper proposes Segment-level Policy Optimization (SPO). SPO introduces a critic-free, Monte Carlo–based advantage estimation at the *semantic segment* level: it employs cut-point–driven chained segmentation and tree-structured advantage propagation over reasoning paths to achieve dynamic, structure-aware segment partitioning; policy updates are then performed segment-wise via probabilistic masking. On GSM8K, SPO outperforms PPO and GRPO by 6–12 percentage points; on MATH500 (with 2K/4K context), it surpasses GRPO by 7–11 points. The implementation is publicly available.

Large language models face significant challenges in multi-step reasoning due to reliance on sparse terminal rewards, which leads to credit assignment difficulties, high gradient variance, and unstable training dynamics. This work proposes Implicit Behavioral Policy Optimization (IBPO), a novel framework that introduces, for the first time, a counterfactual trajectory comparison mechanism. By sampling multiple reasoning trajectories from the same input and leveraging their differences, IBPO implicitly estimates step-level advantages, thereby transforming sparse terminal rewards into fine-grained, step-sensitive learning signals. This approach effectively reduces gradient variance and substantially enhances both training stability and performance ceilings. Experimental results demonstrate that IBPO significantly outperforms existing methods on mathematical and code reasoning benchmarks, while also exhibiting superior continual learning capabilities and a higher upper bound on reasoning performance.

This work addresses the limitation of conventional reinforcement learning methods that rely solely on sparse final rewards for credit assignment, often penalizing effective exploration or rewarding redundant actions. The authors propose TRIAGE, a framework that classifies action segments into four semantic roles—critical progress, beneficial exploration, no progress, or regression—and uses these labels to generate bounded procedural rewards. These intermediate signals are combined with the final task reward to refine policy gradients. Theoretically, TRIAGE achieves optimal segment-level correction using only role labels, substantially reducing advantage estimation error and policy gradient variance. Empirically, it improves task success rates on ALFWorld, Search-QA, and WebShop, while reducing interaction steps by 10.4%–14.8% compared to GRPO, outperforming both scalar procedural rewards and shared-backbone value baselines.

This work addresses the challenge of credit assignment in large language models (LLMs) trained via reinforcement learning, where sparse trajectory-level rewards hinder the identification of critical reasoning steps and lead to low training efficiency. The authors propose OPPO, a novel method that, for the first time, models oracle signals as Bayesian belief updates. By recursively propagating local oracle information along trajectories through Bayesian inference, OPPO dynamically estimates the per-step probability of eventual success, enabling the construction of a token-level advantage function without requiring a value network. The approach unifies self-oracle and teacher-oracle estimators and integrates Bayesian reasoning, token-level credit assignment, and on-policy distillation. Evaluated across seven benchmarks in mathematical, scientific, and code reasoning, OPPO significantly outperforms GRPO, DAPO, and SDPO, achieving gains of 6.0 and 5.2 points on AMC'23 and AIME'24, respectively, with performance improvements monotonically increasing with response length.

This work addresses the limitation of Flow-GRPO, which employs uniform credit assignment and thereby overlooks the heterogeneous contributions of individual steps in the diffusion process, often rewarding suboptimal intermediate states. To overcome this, we propose Stepwise-Flow-GRPO, the first method to implement stepwise credit assignment within flow matching models. Our approach estimates intermediate rewards by leveraging the Tweedie formula based on per-step reward improvements, introduces a gain-based advantage function, and incorporates a DDIM-inspired stochastic differential equation (SDE) to enhance reward quality. By balancing the stochasticity inherent in policy gradients with improved reward accuracy, Stepwise-Flow-GRPO significantly boosts sample efficiency and convergence speed while preserving generation diversity.

This work addresses the limitations of existing verifiable reward-based reinforcement learning methods, which uniformly distribute credit across all tokens and struggle to highlight critical reasoning steps, often relying on external models or ground-truth answers. The authors propose a self-conditioned credit assignment mechanism that operates purely within a reinforcement learning with verifiable rewards (RLVR) setting, leveraging only the model’s own verified trajectories. By employing per-token KL divergence as multiplicative gradient weights, the method enables fine-grained, supervision-free credit assignment. Integrated with the GRPO framework and inspired by self-distillation, the approach includes a theoretical proof that multi-trajectory self-teacher distillation is infeasible. Experiments demonstrate consistent improvements, outperforming GRPO by 8.1% and DAPO by 5.9% on average across five benchmarks—including mathematical reasoning, code generation, and agent tasks—and exhibiting superior out-of-distribution generalization compared to OPD.

This work addresses three structural challenges—channel contamination, granularity mismatch, and cumulative traps—that arise when integrating dense signals with the GRPO framework in process-supervised reinforcement learning for large language models. To resolve these issues, the authors propose PASS, a middleware that reshapes arbitrary scalar step-level process signals through a three-stage mechanism: advantage fusion, value-based chunking, and length normalization, thereby improving credit assignment. PASS is the first approach to systematically identify and simultaneously mitigate all three pathologies, offering paradigm-agnostic and plug-and-play compatibility with diverse process signals such as PRMs and KL distillation. Empirical results demonstrate consistent improvements in pass@1 performance across mathematical reasoning and multi-hop question answering tasks, under two distinct signal paradigms and group normalization operators.

This study addresses the inefficiency of trajectory-level credit assignment in multi-turn agent training, where routine actions dilute gradient signals from critical decisions. It establishes, for the first time, a quantitative √ρ⁻¹ relationship between decision density ρ and the signal-to-noise ratio in policy optimization, demonstrating that non-critical steps at low ρ introduce noise without contributing to returns, thereby severely degrading training efficiency. By constructing controllable environments with tunable ρ, conducting theoretical analysis of trajectory-level policy optimization methods (e.g., GRPO), and modeling gradient variance, the work precisely delineates the applicability boundaries of trajectory-level approaches across high and low decision densities. Experiments strongly corroborate the theoretical predictions (R²=0.999) and reveal a sharp divergence in required training steps as ρ approaches zero.

This work addresses the challenge of credit assignment in long-horizon agents performing multi-turn tool use, where sparse, high-variance, and potentially misleading episodic rewards hinder learning. The authors propose a dense credit assignment method that requires neither auxiliary critics nor process supervision. By modeling trajectories as state transitions at tool-call boundaries, the approach constructs state values using the log-probability of gold answers generated by a frozen reference model and computes dense per-step rewards via temporal difference (TD) learning. A key component—single-step log-ratio TD—automatically suppresses redundant tool invocations. Experiments demonstrate substantial improvements: on BrowseComp-Plus, success rates for Qwen3-4B and Qwen3-30B-A3B rise from 7.2% to 35.6% and from 8.4% to 42.6%, respectively. The method also exhibits strong transferability and faster convergence in open-web tasks.