VeriGate: Verifier-Gated Step-Level Supervision for GRPO

2026-05-28

AI Summary

This work addresses the challenges of sparse supervision signals and inadequate step-level credit assignment in GRPO-based training of reasoning models, which often lead to learning stagnation and insufficient exploration. To overcome these limitations, the authors propose VeriGate, an extension of GRPO that employs a verifier-gated mechanism: it retains the verifier’s dominance when its rewards are reliable but switches to process supervision when the verifier degrades. Furthermore, VeriGate integrates cumulative future rewards with group normalization to enable fine-grained, deception-resistant token-level advantage estimation. Evaluated on the MATH dataset using Qwen2.5-Instruct (1.5B and 7B variants), the method achieves average accuracy improvements of approximately 20% and 12% across six reasoning benchmarks, effectively mitigating zero-gradient failure and reward hacking issues.

Abstract

Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.

Problem

• GRPO
• verifier-based reward
• step-level credit assignment
• sparse supervision
• reasoning models

Innovation

• Verifier-Gated Supervision
• Step-Level Credit Assignment
• Group-Normalized Advantages
• Process Reward Model
• GRPO