Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

2026-06-01

AI Summary

This work addresses the limitations of existing approaches that combine sampling from small models with scoring by stronger models, which often fail to correct early reasoning errors, and process reward models (PRMs) that rely on costly step-level annotations. The authors propose Chunk-Level Guided Generation (CLGG), a novel framework that enables training-free process guidance by leveraging off-the-shelf large language models to compute length-normalized log-likelihood scores for fixed-length reasoning chunks generated by smaller models. A contrastive guided selection strategy (CGS/LGS) dynamically selects high-quality reasoning paths based on these scores. Evaluated on multiple mathematical reasoning benchmarks, CLGG substantially outperforms majority voting—by up to 28 percentage points—and matches or exceeds the performance of trained PRMs under identical computational budgets, while producing shorter reasoning chains.

Abstract

Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels. We propose Chunk-Level Guided Generation, a training-free alternative that uses an off-the-shelf large language model as a process scorer. At each step, a small model samples k fixed-length candidate chunks, while the larger model scores the candidates using likelihoods without generating any text. The selected chunk is committed before the next step, steering generation before errors can propagate. We instantiate this framework with two selection rules: Likelihood-Guided Selection (LGS), which selects the chunk with the highest length-normalized large-model log-probability, and Contrastive-Guided Selection (CGS), which subtracts the small model's log-probability to favor chunks where the large model's preference diverges from the small model's. We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias that persists even after length normalization, and that fixed-length chunks avoid this confound. On GSM8K, MATH, Minerva Math, AMC23, and AIME24 with Qwen2.5-1.5B guided by Qwen2.5-32B and Llama-3.2-1B guided by Llama-3.1-70B, CGS outperforms majority voting by up to 28 pp and, under matched guidance budgets, matches or outperforms Qwen2.5-Math-PRM-72B guided search on most benchmarks without reward-model training. With Qwen2.5-7B guided by Qwen2.5-72B, CGS reaches 81.8% on MATH and 63.6% on Minerva Math at k=16, surpassing majority voting by 4--6 pp. Finally, Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.

Problem

• Mathematical Reasoning
• Process Scoring
• Training-Free Guidance
• Reasoning Path Correction
• Large Language Models

Innovation

• training-free
• process scoring
• chunk-level generation
• contrastive-guided selection
• length bias mitigation