ReCo: Reweighting GRPO Against Distributional Concentration

2026-07-29

AI Summary

This work addresses a critical limitation of Group Relative Policy Optimization (GRPO) in post-training: its tendency to over-rely on high-probability responses from the base model, which diminishes the diversity of reasoning paths and degrades performance under large-k Pass@k evaluation. To mitigate this distributional concentration and collapse during policy optimization, the authors propose ReCo, a novel approach that integrates response-level frequency normalization with a variance-aware, token-level importance ratio. Experimental results demonstrate that ReCo significantly improves Pass@k performance at large k values on both Qwen2.5-Math and Llama-3.1-8B-Instruct, while maintaining comparable effectiveness to GRPO at small k values, thereby enhancing the model’s capacity to explore diverse reasoning trajectories.

Abstract

Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.

Problem

• GRPO
• distributional concentration
• reasoning coverage
• reinforcement learning
• language model post-training

Innovation

• ReCo
• GRPO
• reweighting
• distributional concentration
• importance ratio