APPO: Agentic Procedural Policy Optimization

2026-06-10

AI Summary

This work addresses the limitation of existing reinforcement learning methods for intelligent agents, which perform credit assignment at coarse-grained units—such as tool-call boundaries—and thus struggle to pinpoint the critical intermediate decisions that influence outcomes. To overcome this, the authors propose a fine-grained branching mechanism coupled with a process-level advantage scaling approach, enabling credit assignment at the token level of generated sequences. Branch scores are computed by integrating token-wise uncertainty with policy-induced gains in continuation likelihood, and these scores are combined with program-level advantage scaling to achieve more precise credit allocation. Evaluated across 13 benchmarks, the method yields an average improvement of nearly 4 points while preserving efficient tool usage and behavioral interpretability.

Abstract

Recent advances in agentic Reinforcement Learning (RL) have substantially improved the multi-turn tool-use capabilities of large language model agents. However, most existing methods assign credit over coarse heuristic units, such as tool-call boundaries or fixed workflows, making it difficult to identify which intermediate decisions influence downstream outcomes. In this work, we study agentic RL from two perspectives: \textit{where to branch and how to assign credit after branching}. Our pilot analysis shows that influential decision points are broadly distributed throughout the generated sequence rather than concentrated at tool calls, while token entropy alone does not reliably reflect their impact on final outcomes. Motivated by these observations, we propose \textbf{Agentic Procedural Policy Optimization (APPO)}, which shifts branching and credit assignment from coarse interaction units to fine-grained decision points in the sequence. APPO selects branching locations using a Branching Score that combines token uncertainty with policy-induced likelihood gains of subsequent continuations, enabling more targeted exploration while filtering out spurious high-entropy positions. It further introduces procedure-level advantage scaling to better distribute credit across branched rollouts. Experiments on 13 benchmarks show that APPO consistently improves strong agentic RL baselines by nearly 4 points, while keeping efficient tool-calls and maintaining behavior interpretability.

Problem

• agentic reinforcement learning
• credit assignment
• tool-use
• decision points
• sequence generation

Innovation

• Agentic Reinforcement Learning
• Procedural Policy Optimization
• Fine-grained Credit Assignment
• Branching Score
• Procedure-level Advantage Scaling