Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation

2026-08-03

AI Summary

This study addresses a critical source of evaluation inconsistency in large language models (LLMs) when simulating human perspectives: the conflation of individual response generation (“simulation”) with population-level distribution prediction (“estimation”). For the first time, this work explicitly distinguishes these two tasks and conducts controlled experiments on six base and post-trained models using Pew Research Center’s American Trends Panel data. Model performance is assessed via distribution alignment and demographic structure preservation metrics. Findings reveal that base models excel at simulation, producing individual responses that closely resemble those of real respondents while maintaining demographic representativeness, whereas post-trained models achieve higher accuracy in directly estimating aggregate population distributions. These results highlight the complementary strengths of different model types and provide a principled foundation for task-appropriate model selection.

Abstract

Large language models are increasingly used to simulate human opinions, but prior work reports conflicting results: some studies find promising alignment with human survey data, while others find persona collapse and weak demographic sensitivity. We show that much of this conflict stems from conflating two distinct tasks. We call the first task emulation, in which models generate individual responses that aggregate into a population distribution. We call the second task estimation, in which models directly predict the population distribution. Evaluating six matched base and post-trained models on the Pew American Trends Panel, we find that base models are stronger emulators: they produce response distributions closer to human ground truth and better preserve demographic structure. Post-trained models are stronger estimators, producing more accurate distributional predictions when asked directly. We propose that model selection for human simulation should be guided by whether the task requires generating text or predicting distributions.

Problem

• opinion simulation
• language models
• emulation
• estimation
• demographic sensitivity

Innovation

• emulation
• estimation
• language models
• opinion simulation
• demographic sensitivity