Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posterior

2026-07-21

AI Summary

This work addresses the challenge of efficiently steering pre-trained Transformer models to generate outputs with high expected utility without modifying model parameters or performing backpropagation. To this end, the authors propose Posterior Prompt Tuning (PPT), a novel approach that reframes behavioral guidance as an optimization problem over the latent posterior of Bayesian Filtering Transformers (BFTs). PPT optimizes the distribution of hard prompts using only prior samples and gradient estimates obtained via importance sampling, eliminating the need for forward or backward passes through the model. Notably, a single set of samples suffices to support an arbitrary number of utility functions. Empirical results on Beta–Bernoulli and reinforced urn BFT models demonstrate that PPT effectively guides generation across three distinct utility objectives: inverse cross-entropy, frequency matching, and Dyck language validity.

Abstract

In the \emph{latent posterior model} of transformer behavior, the next-token distribution arises from a posterior over latent predictive models conditioned on the context, mixed to generate continuations. We exploit this model in settings where it is exact, namely Bayes-filtered transformers (BFTs) meta-learned on sequences from a hierarchical prior, to introduce \textbf{Posterior Prefix Tuning (PPT)}, a new method for \emph{eliciting} behavior from a transformer: given a utility function on continuations, find a prompt under which the transformer generates continuations of high expected utility. For a BFT, the elicitation objective factors through the latent posterior, and the gradient of this objective can be estimated from samples of the prior alone. PPT optimizes the parameters of a distribution over hard prompts: it draws prior samples once from the BFT via predictive Monte Carlo (PMC), then estimates the gradient by importance sampling against them. The optimization performs no transformer forward passes and no backpropagation through the transformer, and the prior samples are utility-independent, so a single set of samples drives elicitation against any number of utilities at negligible marginal cost. We validate PPT on Beta--Bernoulli and reinforced urn BFTs across three utility families (reverse cross-entropy, frequency matching, Dyck validity).

Problem

• elicitation
• latent posterior
• transformer
• utility optimization
• prompt tuning

Innovation

• Posterior Prefix Tuning
• Bayes-filtered Transformers
• elicitation without backpropagation
• latent posterior model
• predictive Monte Carlo