Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

2026-08-04

AI Summary

This work addresses the lack of a unified framework in existing test-time scaling methods, which hinders fair comparison of inference algorithms in terms of computational budgets, evaluation metrics, and reproducibility. We propose a formal budgeted inference framework grounded in prefix trees, systematically distinguishing three inference architectures: single-trajectory sequential expansion, leaf-node aggregation, and prefix-level search. For the first time, we establish a three-dimensional analytical framework encompassing structural taxonomy, end-to-end evaluation protocols, and reproducibility standards. By formalizing reasoning-tree modeling, introducing multi-dimensional evaluation profiles, and aligning computational and uncertainty reporting mechanisms, we integrate open-source model ecosystems and validate our approach on benchmarks spanning general knowledge, symbolic reasoning, and competition mathematics. We release over two billion complete reasoning trajectories alongside verifiers and token-level signals.

Abstract

Large language models can solve substantially harder reasoning problems with more inference-time compute. The term "test-time scaling," however, now covers diverse inference algorithms that extend deliberation along a single trajectory, sample completed candidates and aggregate them through voting or verification, or search over unfinished partial states. These algorithms differ in their statistical structure, compute accounting, and failure modes. Treating these procedures as interchangeable under a single scalar "budget," or reporting accuracy without the inference protocol that produced it, makes results difficult to compare across studies. We develop a systematic account of test-time scaling along three axes. First, we formalize test-time scaling as budgeted inference over the implicit prefix tree of an autoregressive model and distinguish three structural regimes: single-trajectory sequential scaling, leaf-level scaling with terminal reduction, and prefix-level scaling. Second, we treat the evaluated object as the entire inference system and develop evaluation principles that separate end-to-end system performance from candidate-bank diagnostics. We introduce an evaluation profile whose coordinates and simple functionals recover or bound common repeated-sampling metrics, and prescribe protocol-matched reporting of compute and uncertainty. Third, we specify reproducibility requirements for inference protocols, distinguishing exact replay from distributional reproducibility and identifying the artifacts needed to support each. We also organize the open-weight reasoning ecosystem by model-side and interface mechanisms, apply these principles to broad-knowledge, symbolic-reasoning, and competition-mathematics benchmarks, and assemble over 2 billion full reasoning traces for release with progressively richer verifier and token-level signals.

Problem

• test-time scaling
• inference protocols
• evaluation
• reproducibility
• reasoning LLMs

Innovation

• test-time scaling
• budgeted inference
• prefix tree
• evaluation protocol
• reproducibility