verifier-guided rl

This work addresses a critical challenge in online policy reinforcement learning with validator-based rewards: sparse sampling can hinder the reinforcement of subsequent successful behaviors even when current objective performance improves, thereby compromising long-term trainability. The study introduces and formally defines the phenomenon of “validator-induced support set reshaping,” demonstrating that endpoint performance gains do not necessarily translate to sustained learnability. It further reveals that policy updates predominantly concentrate on a few initial tokens of model responses. By integrating RLVR, reference policy constraints, routing priors, online distillation, and controlled prompt interventions—alongside token-level distribution analysis and multi-sample evaluation—the approach yields a 6.5% absolute improvement in pass@1 on IFEval for Qwen3-8B-Base within Math-RLVR, albeit at the cost of a 9.8% drop in best@32. Conversely, IF-RLVR reduces best@k performance on mathematical tasks while decreasing downstream reward variance.

This work addresses a critical limitation in existing reinforcement learning post-training methods—the absence of mechanisms to validate the efficacy of policy updates, which often leads to optimization drift or collapse. To mitigate this, the authors propose the PIRL framework, which reframes the optimization objective from immediate reward maximization to cumulative policy improvement across training iterations. Central to this framework is the PIPO algorithm, which introduces, for the first time, a policy improvement feedback mechanism. By employing a sliding window to retrospectively validate historical baselines, PIPO establishes a self-correcting closed-loop optimization process that guarantees each policy update positively contributes to final performance. Empirical evaluations on mathematical reasoning benchmarks demonstrate that PIPO achieves superior stability and performance compared to GRPO and its variants.

Existing process reward models (PRMs) rely on costly step-level human annotations or ground-truth reference solutions, limiting their applicability to domains like mathematical reasoning where gold-standard process annotations are unavailable. Method: We propose SPARK, the first framework for ground-truth-free process-level reward modeling. It employs a generator-verifier collaborative paradigm to produce diverse solution paths, integrates parallel self-consistency scoring, sequence-level meta-critique, and chain-of-thought verification (PRM-CoT) to construct synthetic verification data for fine-tuning a generative PRM, and incorporates format constraints to mitigate reward hacking. Contribution/Results: On ProcessBench, SPARK achieves 67.5 F1—surpassing the ground-truth-supervised baseline (66.4). Across six mathematical reasoning benchmarks, it attains a mean accuracy of 47.4%, significantly outperforming RLVR (43.9%) and establishing the first effective process-supervised reinforcement learning method without reference answers.

This work addresses the inefficiency and vulnerability to reward hacking in existing verifiable-reward-based reinforcement learning approaches for open-ended generation tasks, where ground-truth answers are often unavailable. The authors propose RLVRR, a novel method that extends verifiable rewards from single-point answers to a “reward chain” derived from high-quality reference texts. This framework establishes a dual-track verifiable reward mechanism by evaluating both content fidelity (via keyword retention) and stylistic quality (through large language model validation). By integrating the strengths of reinforcement learning and supervised fine-tuning, RLVRR unifies structured reasoning with open-ended generation in a single training paradigm. Extensive experiments across more than ten benchmarks demonstrate that RLVRR significantly outperforms supervised fine-tuning with ten times more data and state-of-the-art reward models, consistently improving generation quality, generalization, and output diversity.

Existing video generation models exhibit limited capabilities in spatial reasoning and multi-step planning tasks, while reinforcement learning approaches are often constrained by the design of reward functions. This work introduces Group Relative Policy Optimization (GRPO) into flow-based video generation models and proposes two novel reward mechanisms: a multi-component trajectory reward tailored for structured game environments and an embedding-level verifiable reward designed for robotic navigation. The study systematically demonstrates, for the first time, the critical role of verifiable rewards in stabilizing training dynamics. Experimental results show that the proposed method significantly enhances generalization in video-based reasoning, achieving absolute improvements of 29.1% and 51.4% in exact match accuracy over supervised fine-tuning baselines on 3D maze-solving and obstacle-avoidance tasks, respectively.

Traditional reinforcement learning relies solely on sparse, binary final rewards, making it difficult to leverage the rich intermediate feedback available during reasoning processes. This work proposes DistIL, a novel approach that, for the first time, integrates distributed expert feedback with a forward cross-entropy objective within the DAgger framework to enable sequence-level credit assignment. DistIL effectively fuses multi-dimensional signals—such as execution trajectories and tool outputs—through distributional imitation learning, forward KL optimization, and black-box interactions with an expert policy. The method provides theoretical guarantees of monotonic policy improvement and establishes a regret bound. Empirical results demonstrate that DistIL significantly outperforms RLVR and self-distillation baselines across scientific reasoning, code generation, and complex mathematical tasks, achieving substantial gains in Pass@N metrics.

This work addresses the challenges of unfounded reasoning, belief drift, and shortcut behaviors that evade verification in long-horizon language agents trained via reinforcement learning, exacerbated by the absence of process rewards that measure causal contributions of individual reasoning steps. To tackle this, the authors propose CVT-RL, a novel algorithm featuring the first Policy-Conditioned Counterfactual Contribution (PCCC) estimator. By integrating dense verifiable rewards, intervention-effectiveness gating, and controlled interventions—such as deletion and semantic replacement—it enables fine-grained causal credit assignment. The method further incorporates constrained policy gradients, doubly robust advantage estimation, and prefix-based observable belief control. Evaluated across multiple long-context tasks, CVT-RL achieves an average success rate of 78.9% and an evidence F1 score of 82.8%, while reducing automated and human-audited cheating rates to 3.9% and 4.6%, respectively, significantly outperforming baselines and demonstrating strong robustness against adversarial attacks.

Traditional reinforcement learning relies on sparse, all-or-nothing rewards, making it ill-suited for partially verifiable tasks such as multi-requirement instruction following. This work proposes Soft-RLVR, a framework that decomposes task instructions into atomic checklist items and leverages a large language model to assign fine-grained soft rewards based on individual item satisfaction. We further introduce Soft-SVeRL, a self-verification variant wherein the policy model also serves as its own reward validator. For the first time, we formally analyze the trade-off between partial credit assignment and verification noise, and propose an explicit stabilization mechanism to mitigate reward inflation in self-verification. On the IFEval benchmark, our approach achieves an 11.1-point performance gain using only learned verification-based rewards, demonstrating the critical importance of checklist decomposition, verifier quality, and reward stabilization.

This work addresses the limitations of traditional PDE solvers, which rely heavily on expert knowledge and laborious development, as well as existing large language model (LLM) approaches that focus primarily on reasoning optimization while lacking fine-grained feedback on scientific computation accuracy. The authors propose RLVP, a novel framework that introduces, for the first time, a physics-consistency-based continuous reward mechanism combined with a hard constraint on program executability to train LLMs via reinforcement learning for generating high-accuracy solver code. This approach overcomes the shortcomings of conventional binary verification in scientific computing, significantly outperforming both pretrained and supervised fine-tuning baselines across multiple PDE benchmarks. Notably, even smaller models trained with RLVP surpass state-of-the-art prompting strategies of larger models and demonstrate strong zero-shot transfer across PDE types and compositional generalization of numerical modules.

This work addresses the trade-off between training stability and credit assignment fidelity in existing reinforcement learning approaches for large language models: critic-free methods suffer from coarse reward signals, while critic-based methods often exhibit unstable training dynamics. To reconcile these issues, the authors propose a critic-free policy optimization framework that implicitly derives a value function from the optimality conditions of KL-regularized reinforcement learning and constructs a value loss using terminal rewards, thereby enabling fine-grained credit assignment without compromising training stability. By decoupling reward integration from policy updates, the method retains the structural simplicity of critic-free approaches while significantly enhancing credit assignment precision. Empirical results demonstrate consistent and substantial improvements over GRPO on challenging mathematical reasoning benchmarks—including MATH-500, AIME 2024/2025, and OlympiadBench—with notably robust performance in competition-level tasks and under noisy reward conditions.