Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

2026-06-01

AI Summary

This work addresses the challenge of catastrophic forgetting in continual reinforcement learning, where previously acquired solution sets are often lost during training. To mitigate this, the authors propose a retention-aware rehearsal mechanism that explicitly formulates the preservation of correct solutions as an optimization objective. Their approach introduces the “repair window principle” and periodically reintroduces mastered prompts via batch replacement during the pre-rollout phase, achieving memory consolidation with zero additional computational overhead. Implemented within the RLVR framework and leveraging multimodal training with Qwen3-VL and Qwen2.5-Math, the method significantly outperforms GRPO, DAPO, and replay-based baselines across 20 cross-modal benchmarks, demonstrating superior generalization and training stability.

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the ability of large language model, yet headline accuracy gains often conceal a hidden cost: previously solved problems quietly become unsolvable as training proceeds. We frame this phenomenon as \emph{correct-set turnover}, representing the coupled dynamics of solution acquisition and regression over the mastered set. Under this view, retention becomes an explicit optimization target alongside acquisition. We analytically and empirically establish the \emph{repair-window principle}: the cost of restoring a regressed prompt grows sharply with review delay, defining a low-cost window that standard RLVR pipelines fail to exploit. To address this, we propose \textbf{\method{}}, a retention-aware review mechanism that tracks mastered prompts and periodically reintroduces them to \textbf{remind} the model of previous solutions. By utilizing pre-rollout batch replacement, \method{} incurs zero additional rollout overhead. Evaluated across 20 benchmarks spanning image-text, video, and text-only tasks with Qwen3-VL and Qwen2.5-Math, \method{} consistently improves performance over GRPO, DAPO, and replay baselines, demonstrating robust generalizability across modalities and algorithms.

Problem

• correct-set turnover
• retention
• reinforcement learning with verifiable rewards
• solution regression
• forgetting

Innovation

• correct-set turnover
• retention-aware review
• repair-window principle
• RLVR
• zero-overhead rehearsal