Counterfactual Shapley Credit Assignment

2026-07-18

AI Summary

This work addresses the challenge in reinforcement learning of disentangling an agent’s policy contributions from environmental stochasticity in credit assignment. To this end, the authors propose a novel causal inference–based credit assignment framework that introduces counterfactual Shapley values (φ-values) into reinforcement learning for the first time. By integrating φ-values into a policy gradient algorithm—termed φ-PPO—and combining it with Prioritized Trajectory Replay (PTR), the method precisely quantifies the true causal effect of individual actions on final rewards. Evaluated under challenging conditions such as sparse causality, high environmental randomness, and delayed rewards, the approach not only achieves substantially higher attribution accuracy but also maintains optimal policy learning capability, demonstrating superior sample efficiency over existing methods and successfully solving tasks where prior state-of-the-art algorithms fail to converge.

Abstract

The Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents. Existing frameworks, whether relying on temporal contiguity or hindsight-conditioned reward reweighting, frequently fail to attribute properly between an agent's policy (skill) and environmental stochasticity (luck). A principled approach to CAP must isolate the true causal drivers of observed outcomes from spurious correlations and environmental randomness. We introduce Counterfactual Shapley Credit Assignment, a novel framework grounded in causal theory that attributes credit and blame via the Counterfactual Shapley Value ($φ$-value). By redistributing environmental rewards, $φ$-values enhance temporal credit assignment across three critical dimensions: sparse causality, high stochasticity, and delayed rewards, all while preserving the optimal policy. We derive a consistent estimator that computes $φ$-values efficiently, enabling a new class of policy gradient methods, $φ$-PPO, combined with Prioritized Trajectory Replay (PTR). Empirical results demonstrate that $φ$-values align precisely to the ground truth causes of task rewards with superior sample efficiency in challenging environments where prior state-of-the-art methods fail to converge.

Problem

• Credit Assignment Problem
• Reinforcement Learning
• Causal Attribution
• Environmental Stochasticity
• Temporal Credit Assignment

Innovation

• Counterfactual Shapley Value
• Credit Assignment
• Causal Reinforcement Learning
• Policy Gradient
• Temporal Credit Assignment