Verbalized Particle Posterior: Bayesian Inference over Natural Language Hypotheses

2026-07-24

AI Summary

This work addresses the limitations of Verbalized Machine Learning (VML)—notably its lack of uncertainty quantification, instability under single-shot optimization, and susceptibility to catastrophic failures—by introducing a novel Bayesian inference framework operating directly in the space of natural language hypotheses. Treating natural language hypotheses as particles, the method maintains and updates a population of hypotheses via Metropolis-Hastings or Sequential Monte Carlo algorithms, and performs prediction through Bayesian model averaging. It uniquely enables joint modeling of model structure and parameters within a unified natural language hypothesis space, requiring only black-box access to a large language model without relying on logits or gradients, thereby yielding an interpretable Bayesian posterior distribution. Experiments demonstrate that the approach substantially outperforms single-shot VML across regression, classification, and rule discovery tasks, matching or exceeding the performance of ensembled VML baselines while effectively eliminating catastrophic failures.

Abstract

Verbalized Machine Learning (VML) parameterizes a model as a natural-language prompt that an LLM evaluates as f(x; theta). The framework is interpretable, but it commits to a single hypothesis with no measure of uncertainty, and that hypothesis varies substantially across optimization runs on the same data. We propose the Verbalized Particle Posterior (VPP), which treats verbalized learning as a Bayesian inference problem: maintain a population of natural-language hypotheses as particles, update them with Metropolis-Hastings (VPP-MH) or Sequential Monte Carlo (VPP-SMC), and predict by Bayesian model averaging. Both algorithms treat the LLM as a black box, requiring no access to logits or gradients. A distinctive consequence follows. In classical Bayesian learning, model selection sits outside the posterior; in VPP both model structure and parameters share a single language space, and the posterior ranges over both. We evaluate VPP on regression, classification, and rule-discovery benchmarks. It improves over a single VML run on every benchmark and matches or exceeds an oracle-best ensemble of independent VML runs on most, while eliminating the catastrophic single-run failures that VML occasionally produces. Because each particle is a human-readable hypothesis, the posterior is itself something a reader can inspect, seeing in plain text which explanations the data supported and which it ruled out.

Problem

• Verbalized Machine Learning
• Bayesian inference
• natural language hypotheses
• uncertainty quantification
• hypothesis variability

Innovation

• Verbalized Particle Posterior
• Bayesian inference
• natural language hypotheses
• Large Language Models
• model interpretability