Dependency-Aware Reward Shaping for Agentic Reinforcement Learning

2026-10-01

AI Summary

This study addresses the challenges of sparse terminal rewards and task-dependency-agnostic step-level credit assignment in reinforcement learning for large language models. To this end, we propose DARS, a method that constructs predicate dependency graphs to model inter-step relationships, explicitly distinguishing between independent and synergistic operations. Furthermore, DARS transforms automatic annotations into potential-based reward functions to enable fine-grained credit assignment. The proposed approach supports seamless integration with diverse training frameworks. Extensive experiments on benchmarks such as ALFWorld demonstrate that DARS improves success rates by 10%, significantly outperforming existing baselines. The source code has been made publicly available.

Abstract

When training large language models with reinforcement learning, terminal rewards provide little guidance about which steps matter. Common methods for assigning step credit overlook that work built on uncorrected mistakes is wasted while independent work remains valid. With only a final success/failure reward, every step in a failed episode has zero total future reward, even when it made progress. We propose Dependency-Aware Reward Shaping (DARS), which represents task progress as predicates linked by prerequisite relations and assigns step-level credit over the dependency graph. An annotator marks which predicates each step verifies, invalidates, or repairs. Verified predicates are discounted according to graph distance from the nearest broken prerequisite, while independent predicates are unaffected. Repairs update these weights based on any errors that remain; invalidated predicates need re-verification to regain credit. A fixed potential converts these annotations into signed per-step rewards. A common reward and annotation interface allows DARS to integrate with a range of reasoning and agentic training methods, such as GiGPO and ARPO/AEPO, without changing their rollout strategies or optimizers. Across five task families and models from 1.5B to 8B, DARS improves success by up to 10 points over GiGPO trained with the same budget and harness (ALFWorld), raises the WebShop task score and Search-R1 QA accuracy, complements AEPO's entropy-based training on AIME24/25 with a Python interpreter, and exceeds OmniOPD in controlled tool-free reasoning comparisons at 1.7B and 4B. Ablations show that step-level credit, dependency attenuation, and graph topology each contribute. On ALFWorld, a distilled 8B annotator matches the API annotator, enabling DARS to run efficiently without a frontier judge. Code is available at https://github.com/JianhuiWei7/DARS.

Problem

• Reinforcement Learning
• Reward Shaping
• Credit Assignment
• Large Language Models
• Agentic Reasoning

Innovation

• Dependency-Aware Reward Shaping
• Step-level Credit Assignment
• Predicate Dependency Graph
• Agentic Reinforcement Learning
• Reward Shaping