Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimization

2025-08-08

AI Summary

Large language models (LLMs) suffer performance degradation on complex optimization tasks due to inconsistent multi-step planning. To address this, we propose MCTS-OPS, a novel neuro-symbolic framework that introduces Monte Carlo Tree Search (MCTS) into prompt sequence optimization for the first time. It formalizes prompt selection as a reward-driven sequential decision-making process, enabling dynamic exploration and iterative refinement. By tightly integrating LLM-based reasoning with symbolic search, MCTS-OPS enhances logical consistency and improves code generation quality. In network optimization benchmarks, MCTS-OPS achieves a 2–4× improvement in average reward over baseline methods, reduces result variance by 3×, and increases the success rate of finding optimal solutions on challenging instances by approximately 10%. These results demonstrate a substantial enhancement in LLMs’ capability to solve structured, multi-stage optimization problems.

Technology Category

Search and Optimization: Sampling/Simulation-based SearchReasoning under Uncertainty: Stochastic OptimizationConstraint Satisfaction and Optimization: Search

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in code generation and structured reasoning; however, their performance often degrades on complex tasks that require consistent multi-step planning. Recent work has explored combining LLMs with Monte Carlo Tree Search (MCTS), yet existing approaches primarily focus on generating heuristic-based code for optimization or target simpler tasks where correctness alone is sufficient. In this work, we propose MCTS-OPS, a novel neural-symbolic framework that formulates prompt selection as a sequential decision process guided by MCTS. Our method explores and refines multi-step prompt sequences for the goal of improving code generation quality and enhancing the problem-solving capabilities of LLMs in general optimization. Experiments on network optimization show significant improvement over the baselines, both in the success rate of executing the generated code and in the optimization results with the specified objective and constraints (2$sim$4$ imes$ higher reward and 3$ imes$ lower standard deviation). Moreover, it improves the chance of attaining the optimal solution by about 10% of cases, compared to baseline methods in hard problems. These results highlight the promise of combining symbolic planning with LLMs for robust, high-quality code generation in complex domains.

Problem

• Optimizing prompt sequences for LLM-based code generation
• Enhancing multi-step planning in complex optimization tasks
• Improving code execution success and optimization results

Innovation

• Monte Carlo Tree Search guides prompt selection
• Neural-symbolic framework for multi-step prompt sequences
• Improves code generation quality and optimization capabilities