Verifier-Induced Support Reshaping in On-Policy Optimization

2026-07-31

AI Summary

This work addresses a critical challenge in online policy reinforcement learning with validator-based rewards: sparse sampling can hinder the reinforcement of subsequent successful behaviors even when current objective performance improves, thereby compromising long-term trainability. The study introduces and formally defines the phenomenon of “validator-induced support set reshaping,” demonstrating that endpoint performance gains do not necessarily translate to sustained learnability. It further reveals that policy updates predominantly concentrate on a few initial tokens of model responses. By integrating RLVR, reference policy constraints, routing priors, online distillation, and controlled prompt interventions—alongside token-level distribution analysis and multi-sample evaluation—the approach yields a 6.5% absolute improvement in pass@1 on IFEval for Qwen3-8B-Base within Math-RLVR, albeit at the cost of a 9.8% drop in best@32. Conversely, IF-RLVR reduces best@k performance on mathematical tasks while decreasing downstream reward variance.

Abstract

We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget. Across two model families, we study this effect through repeated verifier-scored sampling and bidirectional training on mathematical reasoning and constrained instruction following, including sequential training with the opposite verifier. Math-RLVR raises average instruction-following success but reduces the number of prompts with any successful response under repeated sampling. On IFEval with Qwen3-8B-Base, pass@1 rises by 6.5 percentage points while best@32 falls by 9.8 percentage points, and the same divergence appears across both models and IF benchmarks. Conversely, IF-RLVR shifts math responses from step-by-step openings toward direct answers, lowers best@k across sampling budgets, and reduces reward variation for later Math-RLVR. Token-distribution analyses and controlled opening interventions show that these changes concentrate in the first few response tokens. RLVR mainly reranks openings already available in the base policy, and the selected opening causally affects math searchability. The tested reference-policy constraints, routing priors, and on-policy distillation preserve cross-task support only partially; MathIF and ReasonIF show that marginal gains translate only partly into responses that are both correct and constraint-following. Therefore, endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. Code is available at https://github.com/sylvain-wei/verifier-induced-support-reshaping

Problem

• on-policy optimization
• verifier-induced support reshaping
• reinforcement learning with verifiable rewards
• rewardable support
• trainability

Innovation

• verifier-induced support reshaping
• on-policy reinforcement learning
• verifiable rewards
• response opening reranking
• cross-task support preservation