🤖 AI Summary
This study addresses a critical limitation of large language models (LLMs) in behavioral simulation: while LLMs effectively reproduce descriptive patterns such as attitude distributions, they exhibit substantial biases in estimating causal intervention effects, potentially leading to flawed policy evaluations. For the first time, this work systematically uncovers the disconnect between LLMs’ descriptive fidelity and their causal inference capabilities, identifying key factors that exacerbate this gap. Through natural language prompt–driven simulations, cross-national empirical validation, and attitude–behavior coupling analyses across three country-level datasets, the research demonstrates that LLMs significantly misestimate causal effects for interventions relying on internal subjective experiences and their behavioral outcomes, thereby revealing fundamental limitations in their causal fidelity.
📝 Abstract
Behavioral simulation is increasingly used to anticipate responses to interventions. Large language models (LLMs) enable researchers to specify population characteristics and intervention context in natural language, but it remains unclear to what extent LLMs can use these inputs to infer intervention effects. We evaluated three LLMs on 11 climate-psychology interventions using a dataset of 59,508 participants from 62 countries, and replicated the main analysis in two additional datasets (12 and 27 countries). LLMs reproduced observed patterns in attitudinal outcomes (e.g., climate beliefs and policy support) reasonably well, and prompting refinements improved this descriptive fit. However, descriptive fit did not reliably translate into causal fidelity (i.e., accurate estimates of intervention effects), and these two dimensions of accuracy followed different error structures. This descriptive-causal divergence held across the three datasets, but varied across intervention logics, with larger errors for interventions that depended on evoking internal experience than on directly conveying reasons or social cues. It was more pronounced for behavioral outcomes, where LLMs imposed stronger attitude-behavior coupling than in human data. Countries and population groups appearing well captured descriptively were not necessarily those with lower causal errors. Relying on descriptive fit alone may therefore create unwarranted confidence in simulation results, misleading conclusions about intervention effects and masking population disparities that matter for fairness.