Metropolis-Hastings Dominates Importance Resampling for Policy Composition

2026-10-02

AI Summary

This study addresses the challenges of sampling bias and high retraining costs when combining multiple reward strategies during the inference phase of large language models (LLMs). Grounded in f-divergence theory, we systematically analyze the Metropolis-Hastings (MH) iterative correction algorithm and provide the first proof that, under any computational budget, independence MH strictly dominates sampling importance resampling (SIR) across all convex f-divergences. Furthermore, we theoretically derive an improved error lower bound and quantify the asymptotic error of the proposed approach. Extensive experiments validate the effectiveness of this method in LLM scenarios.

Abstract

Post-training a large language model (LLM) often requires exploring trade-offs between multiple rewards, but retraining for each trade-off is expensive. Decoding-time policy composition allows these trade-offs to be adjusted by combining reward-specific policies at inference time. This composition targets a weighted product of the policies' probabilities over complete responses, but standard implementations combine their next-token probabilities, generally introducing sampling bias. We analyze a known iterative correction based on independence Metropolis-Hastings (MH). Our main result shows that, for every rollout budget, MH produces an output distribution at least as close to the target as sampling-importance-resampling (SIR) with the same budget, as measured by every convex f-divergence. We also derive a lower bound on MH's improvement over the uncorrected decoder in a consensus objective measuring agreement with the supplied policies. We further characterize the correction's sampling error in two asymptotic regimes: when the reward-specific policies approach agreement, and when the log ratio between target and uncorrected-decoder probabilities fluctuates increasingly widely, as can happen for long responses. We complement our analysis with experiments in enumerable and LLM-scale settings.

Problem

• policy composition
• sampling bias
• large language models
• decoding-time alignment
• reward trade-offs

Innovation

• Metropolis-Hastings
• Policy Composition
• Importance Resampling
• Decoding-time Alignment
• f-divergence