Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning

2026-07-21

AI Summary

This work addresses the ambiguity in reward signals and training instability inherent in existing parallel inference methods for large language models, which fail to distinguish individual contributions of diverse reasoning paths. To resolve this, the authors propose the Parallel Shapley framework—the first approach to integrate Shapley values into reinforcement learning for parallel reasoning. By modeling multi-path inference as a cooperative game, the method leverages a generative reward model combined with Monte Carlo sampling to efficiently compute the marginal contribution of each path. This enables fine-grained, interpretable path-level reward allocation, effectively identifying and suppressing uninformative or misleading paths—so-called “free riders.” Evaluated on mathematical reasoning benchmarks, the proposed framework significantly outperforms current state-of-the-art methods, achieving more stable training, superior reasoning performance, and a more transparent reward mechanism.

Abstract

Large Language Models (LLMs) excel at multi-step reasoning, yet current parallel reasoning approaches often fail to distinguish the contributions of individual reasoning paths. Many paths may be redundant, misleading, or even detrimental, but outcome-level rewards assign uniform reward, leading to ambiguous learning signals and unstable training. We propose Parallel Shapley, a reinforcement learning framework that attributes fine-grained, path-level contributions in multi-path reasoning. Treating each path as a player in a cooperative game, we leverage Shapley values to quantify marginal contributions, using a generative reward model to evaluate path utilities and Monte Carlo sampling for efficient approximation. Experiments on mathematical reasoning benchmarks show that Parallel Shapley outperforms existing baselines while providing more stable and interpretable training. Our framework effectively "fishes out the free riders," assigning reward proportionally and improving multi-path reasoning in LLMs.

Problem

• reward attribution
• parallel reasoning
• Shapley values
• free riders
• reinforcement learning

Innovation

• Shapley value
• parallel reasoning
• reward attribution
• reinforcement learning
• large language models