Evaluating LLM-Generated Preference Distributions

2026-09-30

AI Summary

This study addresses the unresolved structural reliability of preference distributions generated by large language models (LLMs) and questions their validity as human preference simulators. We systematically evaluate LLM-generated preference distributions across domains such as aviation and dining, employing repeated sampling, temperature scaling, and prompt perturbation to assess robustness, alongside cross-model comparative analyses. Our findings reveal that while individual LLMs exhibit strong internal consistency, substantial divergence emerges across different models, demonstrating that output variations are primarily driven by model selection rather than prompt engineering. These results challenge the assumption that LLMs possess a universal preference representation and highlight a critical lack of cross-model consensus. Ultimately, this work provides essential empirical evidence advocating for the cautious deployment of LLMs in simulating human preferences.

Abstract

Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settings where real-world data are unavailable. Yet, the structure and reliability of the distributions they produce remain understudied. Here, we systematically analyze LLM-generated distributions of preferences for air travel, restaurants, and consumer products. Encouragingly, all models considered in our analysis exhibit self-coherence, with the most probable outcomes stabilizing rapidly under repeated sampling. At the same time, we observe substantial discordance across both model families and scales, with little consensus even among their most probable outcomes. These patterns hold across nine open-weight models, three choice domains, and show robustness under temperature changes, greedy decoding, and perturbations of prompt and ordering. Our findings indicate that outcomes are influenced more by the choice of model than by the wording of the prompt, challenging the common assumption that sufficiently capable LLMs produce similar preference distributions when used as stand-ins for survey respondents.

Problem

• Large Language Models
• Preference Distributions
• Synthetic Data Generation
• Distribution Reliability
• Model Discordance

Innovation

• Large Language Models
• Preference Distributions
• Self-Coherence
• Probabilistic Generators
• Cross-Model Discordance