Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning

2026-09-22

AI Summary

本文针对大型语言模型在解决复杂推理问题时采用的简单重复采样方法效率低下的问题，提出了一种通过训练小模型生成概念来指导大模型生成答案的新策略，提高了推理准确性。

Abstract

Large language models increasingly tackle hard reasoning problems by spending more test-time compute, yet the dominant strategy remains naive repeated sampling: draw many independent solutions and hope one is correct. Because such sampling explores only through local decoding noise, it tends to produce many near duplicate attempts rather than genuinely different ideas. We ask whether exploration can instead be steered at a semantic level, by first sampling problem specific concepts, hints, or strategies and then conditioning answer generation on them. We refine this into a simple, more exploratory procedure that emits many diverse concepts in a single trajectory, and evaluate it on hard problems where repeated sampling struggles. We then go a step further and make concept generation trainable: a small concept generator is optimized with reinforcement learning so that its concepts maximize the downstream success of a larger, frozen answer generator. On hard mathematical reasoning problems, the trained concept generator substantially improves the answer generator's pass@k over naive repeated sampling at the same answer generation allocation, surpasses concepts drawn from much larger untuned models, and transfers to answer generators it was never trained against, including a model from a different family. A small model can thus be trained into an effective, reusable search policy for a much larger one.

Problem

• large language models
• reasoning problems
• repeated sampling
• exploration
• semantic level

Innovation

• semantic-level exploration
• concept generation
• reinforcement learning
• search policy