Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

2026-07-27

AI Summary

This work addresses the high memory overhead and redundancy in experience replay buffers commonly used in deep reinforcement learning, where conventional compression techniques often introduce bias. The authors propose a novel experience compression method based on n-step sequence endpoints, which constructs a compact buffer by retaining only the initial and terminal states of n-step trajectories. This approach significantly reduces storage requirements while preserving the ability to propagate long-horizon value estimates. By integrating n-step temporal difference learning with an endpoint sampling strategy, the method effectively mitigates systematic bias. Empirical results demonstrate that on the Pinball and Atari 2600 benchmarks, the proposed technique achieves performance comparable to that of standard large-scale replay buffers using only one-tenth of the buffer capacity.

Abstract

Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected $n$-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.

Problem

• experience replay
• recency buffer
• compression
• deep reinforcement learning
• memory efficiency

Innovation

• Endpoint Replay
• Experience Replay
• Buffer Compression
• Deep Reinforcement Learning
• n-step Sequences