Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

2026-08-04

AI Summary

This work addresses a critical limitation in existing unsupervised reinforcement learning with verifiable rewards (RLVR), where policy collapse occurs due to over-reliance on answer-level consensus signals, leading models to reinforce answer tokens rather than improve reasoning capabilities. To mitigate this, the authors propose the OM-GRPO framework, which decouples reward estimation from policy optimization by masking gradients from answer segments while preserving soft consensus-based answer-level rewards. Additionally, OM-GRPO introduces a contrastive reward enhancement mechanism that improves trajectory evaluation without requiring additional samples. Combining group-based relative policy optimization with low-cost pairwise trajectory comparisons, the method significantly outperforms current unsupervised RLVR approaches across multiple reasoning benchmarks and three large language models, matching the performance of supervised training with ground-truth rewards and surpassing majority voting by 4.24 points under test-time training settings.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability. Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples. However, collapse arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning. We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization. OM-GRPO masks gradients on the answer span while retaining answer-level rewards through a soft consensus signal, shifting optimization pressure away from answer tokens. We further introduce Contrast-Augmented Reward, which refines reward estimation via low-cost pairwise comparisons over existing trajectories without additional rollouts. Across diverse reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms existing label-free RLVR methods and matches supervised GT-reward training with stable optimization. This stability is particularly beneficial in the Test-Time Training setting, where OM-GRPO surpasses majority voting by 4.24 points.

Problem

• Reinforcement Learning with Verifiable Rewards
• label-free RLVR
• answer-level consensus
• optimization collapse
• reasoning improvement

Innovation

• outcome-masking
• label-free RLVR
• gradient decoupling
• contrast-augmented reward
• test-time training