Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

2026-04-03

arXiv.org

AI Summary

This work addresses the tendency of current language models to sacrifice reasoning quality for answer accuracy, often producing intermediate steps that are inaccurate, incomplete, or inconsistent. To jointly optimize both prediction accuracy and reasoning fidelity, the authors propose Verifiable Process Supervision (VPS), a post-training framework that supervises structured intermediate assertions. VPS uniquely integrates the verifiability of reasoning steps into a reinforcement learning reward mechanism and employs an error-based adaptive weighting strategy to implicitly form a curriculum that accounts for varying subtask difficulty, thereby discouraging outcome-driven shortcut reasoning. Experiments on chess and mathematical reasoning benchmarks demonstrate that VPS significantly enhances reasoning quality without compromising answer accuracy, reducing worst-case win-rate error by up to 30% and driving reasoning consistency close to saturation levels.

Abstract

Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards typically optimizes only final outcomes, which can lead to a failure mode where task accuracy improves while reasoning becomes less accurate, less complete, or even internally inconsistent. We propose verifiable process supervision (VPS), a post-training framework for verifiable domains that jointly optimizes prediction accuracy and reasoning quality. We first apply supervised fine-tuning to induce a structured reasoning format, enabling syntactic extraction of intermediate claims that are evaluated against ground-truth signals to form process-level rewards. To address the heterogeneous difficulty of reasoning subtasks, we introduce adaptive reward weighting that prioritizes components with the largest remaining errors, creating an implicit curriculum. We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals. While accuracy-only RL improves move accuracy, it sharply degrades reasoning quality, increasing win-rate error by up to 112% and reducing internal consistency by up to 69%. In contrast, VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation. At matched accuracy, judge evaluation also prefers the process-supervised models. A reasoning-space analysis further shows that, without a structured prior, accuracy-only RL converges to budget-dependent shortcuts rather than sound multi-step reasoning. These results show that VPS enables language models to reason both accurately and reliably in verifiable domains.

Problem

• language models
• reasoning quality
• verifiable supervision
• reinforcement learning
• process supervision

Innovation

• verifiable process supervision
• structured reasoning
• adaptive weighting
• process-level reward
• reasoning consistency