BayesPO: Bayesian Prompt Optimization via Parallel-Tempered Gradient-Guided Discrete MCMC

2026-07-17

AI Summary

This work addresses the limitations of existing automatic prompt optimization methods, which often rely on heuristic search and struggle to balance task performance with linguistic fluency. The authors formulate prompt optimization as a Bayesian posterior sampling problem over discrete prompt tokens, constructing an energy function that integrates task-specific likelihood with a language model prior. To efficiently explore this discrete space, they propose a novel discrete MCMC sampling strategy—based on Gibbs-with-Langevin—that combines gradient information with parallel tempering to achieve a balance between global exploration and local refinement. Experiments on Qwen2.5 demonstrate that the method improves the average accuracy of APE prompts to 63.23% across 24 instruction induction subtasks and effectively escapes local optima in tasks such as poetry generation, confirming its effectiveness and generalizability.

Abstract

Prompt optimization adapts large language models (LLMs) without updating model parameters, but many automatic prompt optimizers remain heuristic search procedures over candidate instructions. This paper studies prompt optimization as Bayesian posterior sampling over discrete prompt tokens. We define a posterior distribution by combining a task likelihood term, which rewards prompts that explain input-output examples, with a language-model prior, which favors fluent instructions. This converts prompt optimization into an energy-based posterior sampling problem, for which gradients can be used to guide discrete Markov chain Monte Carlo (MCMC) proposals over vocabulary tokens. We refer to our framework as BayesPO, short for Bayesian Prompt Optimization. In this paper, BayesPO is instantiated with Markov chain Monte Carlo: it uses a Metropolis-Hastings corrected Gibbs-with-Langevin (GwL) proposal and integrates parallel tempering for global exploration of rugged LLM-induced energy landscapes. The concrete sampler further adapts the GwL sampler to the practical constraints of non-weight-tied LLM embeddings. Experiments with Qwen2.5 models show that the sampler discovers semantically meaningful prompts on diagnostic tasks, that parallel tempering helps escape a local optimum in a poetry completion task, and that post-optimizing APE prompts on 24 instruction-induction subtasks improves average accuracy from 60.04% to 63.23%. The study also reveals two main limitations: energy minimization may overfit small optimization sets, and the current sampler remains computationally expensive. These findings position Bayesian prompt sampling as a principled post-optimization tool and point to a promising direction for probabilistic prompt optimization.

Problem

• prompt optimization
• Bayesian inference
• discrete MCMC
• large language models
• energy-based sampling

Innovation

• Bayesian prompt optimization
• discrete MCMC
• gradient-guided sampling
• parallel tempering
• energy-based modeling