DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

2026-08-07

AI Summary

This work addresses the challenge of fine-grained credit assignment in training coding agents, where interleaved code modifications obscure responsibility attribution. To resolve this, the authors propose a critic-free reinforcement learning approach that parses the internal structure of code diffs, decomposing multi-turn coding interactions into thought-action steps. By introducing an “aggregatability score” to guide the segmentation of sub-diffs, the method constructs semantically coherent and aggregatable credit units. This enables precise backpropagation of advantage signals to the corresponding response tokens, facilitating fine-grained policy optimization. Evaluated on long-horizon coding and reasoning benchmarks, the proposed framework, verl-code, substantially outperforms existing RL methods—achieving over a 10% performance gain on Qwen2.5-7B-Coder and significantly narrowing the gap with larger models.

Abstract

Reinforcement learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for training coding agents, where the execution feedback from compilation and tests provides objective verification. However, unlike agent tasks, coding agents face a unique and finer-grained credit assignment challenge: at each step, coding actions simultaneously pack varying changes into different regions of a code version, which makes the contribution of independent change indistinguishable. Existing RLVR methods mostly leverage the outcome reward or step-level reward, which fails to dive into a code diff and makes unique properties of coding actions invisible to training. In this paper, we propose Diff-in-Diff Policy Optimization (DiDPO), a critic-free RL method that constructs fine-grained credit units directly from the structure of code diffs. DiDPO organizes multi-turn coding interactions into multiple thought--action steps and discovers code diffs across sampled trajectories. It then selects anchors by aggregating highly similar sub-diffs split from each whole diff by our ``groupability score'', which provides the splitting schema that optimally balances the semantic scope of anchors and the group mass they may form. Finally these anchors form advantage groups and project the diff-level advantage back to individual response tokens. Experiments on long-horizon coding and reasoning benchmarks show that DiDPO significantly outperforms strong agentic RL baselines. On Qwen2.5-7B-Coder, DiDPO exceeds comparable methods by over 10\% and narrows the gap with far larger models, offering a principled framework for fine-grained credit assignment in coding agent training. We also open-source verl-code, an agentic rl codebase that supports various RL methods and coding benchmarks.

Problem

• credit assignment
• coding agents
• code diffs
• reinforcement learning
• fine-grained reward

Innovation

• fine-grained credit assignment
• code diff
• reinforcement learning
• DiDPO
• groupability score