Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

2026-07-21

AI Summary

This work addresses the "cliff" problem in reinforcement learning for large language models on challenging reasoning tasks, where sparse or absent reward signals hinder effective learning. The authors propose Off-Context GRPO, a novel approach that introduces an “off-context” rollout mechanism to generate non-zero reward samples by leveraging privileged information—such as solution prefixes—as guidance during rollouts. To preserve policy consistency with the original unguided distribution, they design an importance-corrected objective that pulls policy updates back toward the target distribution while retaining the benefits of guided exploration. Evaluated on standard mathematical reasoning benchmarks, Off-Context GRPO improves average accuracy by 3.9 percentage points (a relative gain of 13.8%) over the original GRPO, with negligible additional computational overhead.

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives \textit{zero} learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts \textit{off-context}: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.9\% absolute improvement (13.8\% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.

Problem

• reinforcement learning
• reasoning
• privileged information
• learning signal
• hard problems

Innovation

• Off-Context Rollouts
• Privileged Information
• Importance Correction
• Reinforcement Learning with Verifiable Rewards
• GRPO