Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents

2026-06-03

AI Summary

This work addresses the challenge of optimizing black-box large language model (LLM) agents, which are inaccessible for parameter-level reinforcement learning due to API-only access. Leveraging the equivalence between reinforcement learning and Bayesian inference, the authors propose a test-time optimization method that operates without modifying model parameters. The approach employs sequential Monte Carlo to sample trajectories from the posterior over optimal policies, guided by a learned value function to improve sampling efficiency. This framework constitutes the first principled, reinforcement learning–inspired test-time optimization strategy for black-box LLM agents. Evaluated on three environments in AgentGym, the method significantly outperforms prompt engineering baselines and, with increased test-time computation budgets, even surpasses the performance of trainable methods such as GRPO.

Abstract

LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents. Code is available at https://github.com/layer6ai-labs/Agentic-Monte-Carlo

Problem

• black-box agents
• reinforcement learning
• large language models
• test-time optimization
• parameter-free control

Innovation

• Agentic Monte Carlo
• Black-box LLM agents
• Reinforcement Learning
• Sequential Monte Carlo
• Bayesian inference