experience replay

This work systematically investigates the underutilized potential of experience replay in reinforcement learning-based post-training of large language models, challenging the prevailing assumption that fresh online data generation is indispensable. By carefully balancing data staleness, sample diversity, and computational cost, the authors design an efficient replay buffer mechanism that effectively substitutes strict online sampling. Their approach demonstrates, for the first time, that experience replay can significantly reduce inference-time computational overhead while maintaining or even improving model performance and effectively preserving policy entropy. This finding offers a compelling alternative to costly online data collection, suggesting that strategic reuse of historical interactions can sustain training efficacy without compromising behavioral diversity or learning stability.

Offline-to-online reinforcement learning (O2O RL) suffers from an imbalance in trade-offs between offline data and online experience: fixed mixing ratios fail to simultaneously ensure early training stability and long-term performance improvement. To address this, we propose the Adaptive Replay Buffer (ARB), a training-free mechanism that introduces the first trajectory-level, learning-agnostic on-policyness metric based on policy consistency. ARB enables real-time, lightweight dynamic adjustment of sampling weights within the replay buffer as the policy evolves. It relies solely on behavior cloning error and action-output alignment—requiring no additional model training or gradient computation—and is fully compatible with mainstream O2O algorithms (e.g., CQL, BEAR). Evaluated on the D4RL benchmark, ARB significantly mitigates early performance degradation and improves final returns by an average of 18.7%. It exhibits strong generalization across tasks and incurs negligible computational overhead.

Catastrophic forgetting remains a fundamental challenge in continual learning. Method: This paper systematically investigates memory sampling strategies for experience replay, challenging the conventional default assumption of uniform sampling. We propose 50 non-uniform sampling strategies based on randomized weighted probability distributions and evaluate them within standard replay frameworks, integrated with mainstream continual learning methods (e.g., EWC, LwF) and benchmarks (Split-CIFAR100, PermutedMNIST). Contribution/Results: Experiments across diverse buffer sizes, model architectures, and task sequences consistently identify at least one non-uniform strategy that significantly outperforms uniform sampling—demonstrating its robust and general effectiveness. Crucially, we provide the first empirical evidence that adaptive replay strategies play a pivotal role in mitigating forgetting. Our work establishes a new paradigm for experience replay design and delivers a reproducible, optimization-guided pathway for improving replay-based continual learning systems.

Sample replay is widely employed in continual learning to mitigate catastrophic forgetting of old tasks, yet its efficacy remains poorly understood. Method: We theoretically analyze replay in an idealized, noiseless, overparameterized linear regression setting and conduct empirical validation via SGD-trained neural networks on standard benchmarks. Contribution/Results: We establish, for the first time, that replay can *worsen* both worst-case and expected forgetting—demonstrating non-monotonic and even detrimental effects. The core mechanism is a coupling between task subspace geometry and replay sample selection: when replayed samples deviate from the principal directions of old-task subspaces, they amplify parameter drift and induce negative transfer. Our theoretical forgetting bounds and extensive experiments consistently reproduce and confirm this phenomenon. This work challenges the assumption that replay is universally beneficial, revealing its effectiveness to be critically contingent on task geometry and replay strategy—providing essential theoretical guidance and caution for designing robust replay mechanisms in continual learning.

Catastrophic forgetting remains a fundamental challenge in continual learning. This work investigates sample-level forgetting sensitivity and identifies a strong correlation between learning order and forgetting severity: samples learned earlier exhibit greater resistance to forgetting. Motivated by this finding, we propose the “Goldilocks” sampling principle—selecting only moderately learned samples for rehearsal while excluding those learned too quickly or too slowly. We further design a training-dynamics-based model to estimate per-sample learning speed and integrate it into a dynamic buffer update mechanism. Our approach is lightweight and seamlessly compatible with mainstream rehearsal methods (e.g., ER, A-GEM). Extensive experiments on Split-CIFAR10/100 and Split-ImageNet demonstrate state-of-the-art performance: average accuracy improves by 2.1–3.7%, and forgetting rates decrease significantly. To our knowledge, this is the first work to quantitatively establish the relationship between learning timing and forgetting, introducing a novel sample-aware paradigm for continual learning.

This work addresses catastrophic forgetting in continual learning under non-stationary data streams by proposing the COLD framework, which introduces, for the first time, the Drift-Plus-Penalty stochastic optimization method from control theory into this domain. COLD formulates forgetting as a controlled dynamic process, employing virtual queues to track performance deviations on historical tasks and jointly minimizing the current task loss and queue drift at each optimization step. This mechanism explicitly governs the stability-plasticity trade-off. The framework provides theoretical guarantees on stability and convergence, and achieves significantly superior performance over state-of-the-art methods on standard benchmarks, enabling controllable and efficient suppression of catastrophic forgetting.

This work addresses the challenge of catastrophic forgetting and misalignment in continual instruction tuning, where fixed replay ratios fail to adapt to dynamic task distributions. The authors propose PROXYMIX, a novel framework that leverages the “forgetting mirror” hypothesis—empirically validated for the first time—which posits that the relative forgetting sensitivity across tasks remains consistent across model scales. By training a dynamic replay controller on a small proxy model, PROXYMIX transfers this policy to large models without requiring knowledge of future tasks. The controller constructs its state from normalized validation loss and its temporal dynamics, then adaptively blends old and new data via a mask-based mixing mechanism. Evaluated on five sequential instruction-tuning benchmarks with LLaMA-3-8B, PROXYMIX improves average accuracy by 3.4 points, reduces final forgetting by 3.5 points, enhances safety by 5.8 points, and achieves these gains at only 1/50th the policy learning cost of Oracle Target RL.

This work addresses the instability and low sample efficiency in Generalized Reinforcement Learning with Policy Optimization (GRPO) for large language model post-training, which stem from rapid policy drift causing experience replay samples to become outdated. To mitigate this issue, the authors propose a rollout-level experience replay mechanism tailored for GRPO, integrating rollout-based storage and sampling, advantage-weighted prioritized replay, a maximum step-age (τ_max) eviction strategy, and the construction of fresh anchor batches. Experiments on three Qwen3-Base model scales across five mathematical benchmarks demonstrate that the proposed method significantly outperforms both standard GRPO and naive replay baselines, achieving an average accuracy gain of 4.35 percentage points and an improvement of 0.579 in the AES efficiency metric for the 4B model.

This work addresses a critical vulnerability in existing experience replay mechanisms for continual learning: their susceptibility to stealthy attacks that manipulate replay sample selection without altering the data itself. We demonstrate for the first time that an adversary with access only to replay indices can severely degrade model performance by skewing the class distribution of replayed samples. To this end, we propose an audit-aware stealthy attack framework that balances invisibility and impact. Our method estimates per-class utility via lightweight metrics—either EMA loss or confidence—and optimizes the replay distribution using projection techniques based on KL divergence (via exponential tilting) or total variation distance (via mass redistribution). A sliding-window scheduler further adapts the attack to rolling audits. Experiments across multiple continual learning benchmarks show significant drops in final accuracy and backward transfer, with the KL-based variant achieving high destructiveness while remaining highly evasive under diverse auditing mechanisms.

This work addresses a critical gap in continual learning research: while most existing methods focus on mitigating catastrophic forgetting, they largely overlook the conditions under which forward transfer—where knowledge from past tasks benefits new ones—can be effectively realized. The paper introduces, for the first time, a systematic three-condition framework that characterizes when forward transfer is feasible and proposes Transfer-Selective Replay (TSR), a novel method that leverages a zero-training-overhead task signature mechanism to automatically identify and replay only those historical samples beneficial to the current task. TSR integrates knowledge distillation to preserve performance on previous tasks while explicitly promoting forward transfer as a first-class objective. Experiments demonstrate that TSR significantly enhances forward transfer across both homogeneous and heterogeneous task sequences and consistently outperforms existing replay-based baselines, especially under limited replay budgets.