Non-Uniform Memory Sampling in Experience Replay

2025-02-16

AI Summary

Catastrophic forgetting remains a fundamental challenge in continual learning. Method: This paper systematically investigates memory sampling strategies for experience replay, challenging the conventional default assumption of uniform sampling. We propose 50 non-uniform sampling strategies based on randomized weighted probability distributions and evaluate them within standard replay frameworks, integrated with mainstream continual learning methods (e.g., EWC, LwF) and benchmarks (Split-CIFAR100, PermutedMNIST). Contribution/Results: Experiments across diverse buffer sizes, model architectures, and task sequences consistently identify at least one non-uniform strategy that significantly outperforms uniform sampling—demonstrating its robust and general effectiveness. Crucially, we provide the first empirical evidence that adaptive replay strategies play a pivotal role in mitigating forgetting. Our work establishes a new paradigm for experience replay design and delivers a reproducible, optimization-guided pathway for improving replay-based continual learning systems.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Continual learning is the process of training machine learning models on a sequence of tasks where data distributions change over time. A well-known obstacle in this setting is catastrophic forgetting, a phenomenon in which a model drastically loses performance on previously learned tasks when learning new ones. A popular strategy to alleviate this problem is experience replay, in which a subset of old samples is stored in a memory buffer and replayed with new data. Despite continual learning advances focusing on which examples to store and how to incorporate them into the training loss, most approaches assume that sampling from this buffer is uniform by default. We challenge the assumption that uniform sampling is necessarily optimal. We conduct an experiment in which the memory buffer updates the same way in every trial, but the replay probability of each stored sample changes between trials based on different random weight distributions. Specifically, we generate 50 different non-uniform sampling probability weights for each trial and compare their final accuracy to the uniform sampling baseline. We find that there is always at least one distribution that significantly outperforms the baseline across multiple buffer sizes, models, and datasets. These results suggest that more principled adaptive replay policies could yield further gains. We discuss how exploiting this insight could inspire new research on non-uniform memory sampling in continual learning to better mitigate catastrophic forgetting. The code supporting this study is available at $href{https://github.com/DentonJC/memory-sampling}{https://github.com/DentonJC/memory-sampling}$.

Problem

• Non-uniform sampling in experience replay
• Mitigating catastrophic forgetting
• Improving continual learning performance

Innovation

• Non-uniform memory sampling
• Adaptive replay policies
• Mitigate catastrophic forgetting