On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference

2025-12-04

AI Summary

This work identifies the theoretical suboptimality of mainstream test-time computation (TTC) methods—such as Best-of-N (BoN) and sequential refinement—arising from their fixed sampling strategies and static context updating mechanisms. To address this, we propose Reward Filtering (RF), a novel TTC framework that dynamically evaluates candidate sequences via reward estimation during inference, retains only high-reward candidates in the context, and integrates hybrid reference selection with dynamic context updating for efficient sequence optimization. RF theoretically approaches the optimal decision frontier, offering stronger performance guarantees than BoN. Experiments across benchmarks—including GSM8K, MMLU, and HumanEval—demonstrate that RF consistently outperforms existing TTC methods, delivering stable, robust, and statistically significant gains. Our approach establishes a more principled, provably superior paradigm for large language model test-time inference.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Learning Preferences or Rankings
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Test-time compute (TTC) has become an increasingly prominent paradigm for enhancing large language models (LLMs). Despite the empirical success of methods such as best-of-$n$ (BoN) sampling and sequential revision, their fundamental limits remain unclear. We address this gap by analyzing a mixture-of-reference policy model and proving that standard BoN is inherently suboptimal. To move closer to the optimal frontier, we study reward-filtered sequential inference, a simple procedure that selectively incorporates only high-reward generations into the context. This mechanism concentrates computation on superior policy candidates and suppresses inferior ones. On the theoretical side, we show that reward-filtered sequential inference yields strictly stronger guarantees than standard TTC paradigms. On the empirical side, we evaluate such an inference strategy across diverse benchmarks and observe consistent improvements over widely used approaches, demonstrating the practical effectiveness of our framework.

Problem

• Analyzes limits of test-time compute for LLM enhancement.
• Proposes reward-filtered sequential inference to improve performance.
• Demonstrates theoretical and empirical gains over existing methods.

Innovation

• Reward-filtered sequential inference improves LLM performance
• Selective high-reward generation integration optimizes computation
• Theoretical and empirical gains over standard test-time compute