Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe

2026-07-28

AI Summary

This study addresses a critical limitation in instruction-tuned language models: despite accurately capturing the distribution of human responses, they fail to sample effectively from it, instead collapsing to a single deterministic output—a phenomenon the authors term the “KNOWS/DOES” split. This degeneracy stems from alignment training inducing pathological decoding behavior. The work presents Prompt-Perturbed Argyle (PPA), a novel method that mitigates sampling collapse without additional computational overhead. Through comprehensive analyses—including logits inspection, comparisons between base and fine-tuned models, and evaluation of both distributional fidelity and sampling behavior—the authors demonstrate PPA’s efficacy. A single model call reduces sampling error by over 50%, and PPA further decreases this error by an additional 21%.

Abstract

Silicon sampling uses language models as proxies for human survey respondents, treating each model call as an independent draw from the persona's response distribution. We show this draw does not exist: instruction-tuned models do not sample from distributions, they collapse to a single output. The same persona on the same question returns the same answer on more than half of items in a public-opinion benchmark. The collapse is sharp: the model's internal probabilities concentrate on a single option, and the failure is substantially amplified by instruction tuning: across three model families with materially different post-training pipelines, every instruction-tuned model fails on every task we test, while base models fail far less often. Strikingly, the same model that cannot sample from a distribution can describe it accurately in a single call. We call this gap the KNOWS/DOES split, and trace it to a degenerate sampling primitive visible in the logits and induced by alignment training. Exploiting this split, asking the model to describe the response distribution in one call more than halves the error against human survey data compared to persona aggregation. For applications that require per-persona outputs, we propose Prompt-Perturbed Argyle (PPA), which reduces the same error by 21% at no added cost.

Problem

• instruction tuning
• distribution sampling
• language models
• response diversity
• alignment training

Innovation

• instruction tuning
• sampling collapse
• KNOWS/DOES split
• language model alignment
• distribution description