Problem
Research questions and friction points this paper is trying to address.
Solution Mode Collapse
Reinforcement Learning with Verifiable Rewards
Solution Diversity
Language Models
Measuring and Mitigating Solution Mode Collapse in RLVR
🤖 AI Summary
This study addresses the issue that Reinforcement Learning with Verifiable Rewards (RLVR) is prone to mode collapse, leading to homogenized problem-solving strategies. To this end, we introduce ModeBench, the first benchmark for evaluating multiple correct solutions, which empirically demonstrates that RLVR exacerbates the loss of solution diversity. Furthermore, we propose Re:Max, a method that leverages a mode-aware reward mechanism and a replay buffer to ensure balanced training across all valid solution modes. Experimental results show that Re:Max significantly improves task success rates while effectively enhancing solution diversity. This work establishes a novel paradigm for mitigating mode concentration in frontier large language models.
📝 Abstract
A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces. A solution will earn the same reward whether it is the thousandth copy of a familiar answer or one the model has never produced before. Yet, there is potential value in having the model retain multiple correct solutions as it is trained. For instance, multiple modes may give users a choice and provide problem-solving strategies that improve overall model performance. Here, we introduce ModeBench, a benchmark of multi-solution tasks in which the verifier returns both correctness and mode discovered. We then use ModeBench to measure how solution diversity changes under RLVR post-training. We find that RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves, and moreover, that frontier models are already highly concentrated. We then introduce our solution, Re:Max, which stores one verified example per discovered mode in a replay buffer and trains on those stored modes uniformly. A solution found once is, therefore, practiced as often as one found repeatedly. Across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Solution Mode Collapse
RLVR
ModeBench
Re:Max
Replay Buffer
🔎 Similar Papers
No similar papers found.
