SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute

2026-07-30

AI Summary

This work addresses the inefficiency of uniform computation allocation in existing language models during reasoning and the impracticality of deploying external-feedback-based verification mechanisms. The authors propose Self-Verification Refinement (SVR), a novel framework that leverages self-verification as an internal control signal to enable adaptive, unsupervised computation allocation at test time. SVR jointly optimizes answer correctness and confidence estimation through multi-round reinforcement learning, dynamically deciding whether to continue reasoning. It employs a GRPO algorithm trained with a composite reward combining correctness, calibration-aware self-verification, and stop-state incentives. Evaluated on seven mathematical reasoning benchmarks using Qwen3.5-2B, SVR achieves an average accuracy of 0.563 in just 2.99 reasoning rounds, significantly outperforming standard GRPO, multi-round baselines, and fixed-budget oracle-guided approaches.

Abstract

Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback. We introduce Self-Verifying Refinement (SVR), an oracle-free multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy. At each turn, the model produces a solution together with a discrete correctness verdict and a confidence score; it retains the current answer only when the verdict is Correct and confidence exceeds a threshold, and otherwise continues refinement using its own self-verification. Ground-truth correctness is used only to construct training rewards and is never exposed to the policy through refinement prompts or required at inference. SVR is trained with GRPO on fixed-horizon trajectories using rewards that promote solution correctness, calibration-aware self-verification, and stop-ready correct states; adaptive stopping is activated only at inference. On seven mathematical reasoning benchmarks with Qwen3.5-2B, SVR achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average. In the evaluated complete-system comparison, it exceeds standard GRPO, strong multi-turn baselines, and a fixed-budget oracle-guided score-feedback reference while requiring substantially fewer turns than fixed ten-turn inference. These results demonstrate that learned self-verification can serve as an effective internal control signal for answer retention and adaptive test-time compute allocation.

Problem

• adaptive test-time compute
• self-verification
• reasoning
• compute allocation
• language models

Innovation

• Self-Verifying Refinement
• test-time compute scaling
• reinforcement learning
• adaptive stopping
• calibration-aware verification