Bootstrapped Reward Shaping

2025-01-02

AI Summary

To address slow convergence in reinforcement learning under sparse rewards, this paper proposes a bootstrapped potential-based reward shaping (V-PBRS) method grounded in state-value function estimation. Unlike conventional approaches, V-PBRS eliminates the need for handcrafted potential functions by dynamically constructing them directly from the current estimate of the state-value function. It rigorously preserves the optimality of the original policy and provides theoretical convergence guarantees. By integrating the potential function into both Q-learning and DQN frameworks—leveraging Monte Carlo or temporal-difference online estimation mechanisms—V-PBRS achieves stable and adaptive reward shaping. Empirical evaluation on the Atari benchmark demonstrates significantly accelerated training convergence, validating its effectiveness, robustness, and generalization capability in high-dimensional visual environments.

Technology Category

• Machine Learning: Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations,"potential-based reward shaping"(PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required"potential function"must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a"bootstrapped"method of reward shaping, termed BSRS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite.

Problem

• Sparse Rewards
• Reinforcement Learning
• Slow Learning

Innovation

• BSRS Method
• Potential Function
• Accelerated Learning in Atari Games