Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries

2026-10-05

AI Summary

This study addresses the high computational costs of test-time training for large language models and the credit assignment challenges arising from coupled policy and implementation. We propose Guidance-TTT, a framework that introduces a novel guidance-execution decoupled architecture. Specifically, it freezes a large model to handle code implementation while exclusively training a small model to optimize high-level decision-making policies, thereby restricting test-time learning to short-horizon decision sequences. Furthermore, we design an adaptive group-relative reinforcement learning objective that leverages verifier feedback to enable online policy updates. Experimental results demonstrate that, in offline settings, this framework outperforms state-of-the-art methods across four major domains, including combinatorial optimization and machine learning, while significantly reducing computational overhead and enhancing exploration efficiency.

Abstract

Open-ended scientific discovery often requires repeatedly proposing and evaluating candidate solutions. LLM-based systems can support this process by generating and refining executable solutions from verifier feedback. Methods such as TTT-Discover use test-time training (TTT) to update the solution-generating LLM from verifier feedback, adapting its generation policy to improve subsequent proposals on the target problem. However, this becomes expensive when reliable execution requires a large model, since training must maintain gradients, optimizer states, and policy statistics while repeatedly generating long, structured outputs. It also complicates credit assignment: outcome-level verifier feedback must jointly evaluate the high-level strategy and its low-level implementation. In this work, we introduce Guidance-TTT, which separates these roles. A compact guidance model is trained at test time to propose high-level strategic changes, while a frozen execution model implements them as complete executable solutions. At each step, the system selects a promising previously discovered solution, proposes a change, executes and verifies it, and updates only the guidance model using an adaptive group-relative RL objective. This concentrates test-time learning on short strategic decisions while retaining the implementation capability of a substantially stronger model without adapting it. Without web access, Guidance-TTT produces strong solutions across four distinct domains: combinatorial optimization (Polyomino Packing), heuristic programming (AHC058), machine learning (Lasso), and GPU kernel optimization (TriMul). Across these tasks, it outperforms the best solutions reported in prior work while remaining competitive with state-of-the-art results on public online leaderboards. Code is available at https://github.com/Human-Agent-Society/reef/tree/guidance-ttt-support.

Problem

• Open-ended scientific discovery
• Test-time training
• Credit assignment
• Large language models
• Computational cost

Innovation

• Test-time training
• Guidance-execution decoupling
• Group-relative reinforcement learning
• Open-ended scientific discovery
• Small model adaptation