Reward Prediction Error Prioritisation in Experience Replay: The RPE-PER Method

2025-01-30

AI Summary

To address low sampling efficiency in experience replay—particularly slow convergence under dynamic environments and limited data—this paper introduces a novel priority sampling mechanism based on Reward Prediction Error (RPE). Drawing inspiration from neuroscience, we are the first to incorporate RPE into deep reinforcement learning by designing an Echo-Modulated Critic Network (EMCN) that jointly predicts Q-values and immediate rewards; the resulting RPE serves as a dynamic, unbiased priority signal. Our method integrates seamlessly into off-policy Actor-Critic frameworks without requiring auxiliary labels or trajectory relabeling. Evaluated on multiple continuous-control benchmark tasks, it significantly accelerates convergence compared to standard Prioritized Experience Replay (PER) and Hindsight Experience Replay (HER), achieving an average 12.7% improvement in final performance while enhancing both sample efficiency and training stability.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reinforcement Learning algorithms aim to learn optimal control strategies through iterative interactions with an environment. A critical element in this process is the experience replay buffer, which stores past experiences, allowing the algorithm to learn from a diverse range of interactions rather than just the most recent ones. This buffer is especially essential in dynamic environments with limited experiences. However, efficiently selecting high-value experiences to accelerate training remains a challenge. Drawing inspiration from the role of reward prediction errors (RPEs) in biological systems, where they are essential for adaptive behaviour and learning, we introduce Reward Predictive Error Prioritised Experience Replay (RPE-PER). This novel approach prioritises experiences in the buffer based on RPEs. Our method employs a critic network, EMCN, that predicts rewards in addition to the Q-values produced by standard critic networks. The discrepancy between these predicted and actual rewards is computed as RPE and utilised as a signal for experience prioritisation. Experimental evaluations across various continuous control tasks demonstrate RPE-PER's effectiveness in enhancing the learning speed and performance of off-policy actor-critic algorithms compared to baseline approaches.

Problem

• Reinforcement Learning
• Experience Replay
• Efficient Sampling

Innovation

• Reward Prediction Errors
• Prioritized Experience Replay
• Reinforcement Learning Acceleration