🤖 AI Summary
This study investigates whether coarse-grained marginal validation alone suffices to support large language models (LLMs) as surrogate participants in human behavioral research. We construct a fixed panel of 16 lightweight personality-conditioned GPT-4 instances and evaluate their marginal alignment with human data in repeated games, systematically analyzing the impact of prompts, phrasing, and labeling on behavioral outputs. Methodologically, we integrate a fixed-panel design, symmetric Dirichlet sensitivity analysis, finite-opportunity plug-in estimation, and exact gating tests, alongside a verifiable reproduction framework that requires no real-time API calls. Results show that three of four game units meet preregistered marginal criteria; prompt variations account for 47%–96% of behavioral variance; and minor wording adjustments increase cooperation rates from 0/40 to 37/40. While marginal matching is achievable, treatment effect estimates remain imprecise, revealing methodological limitations including household-level error, interdependence, and boundary uncertainty.
📝 Abstract
Large language models are increasingly used as synthetic research participants and are often validated by whether their marginal responses resemble human data. We study a fixed panel of sixteen lightweight persona-conditioned GPT-4.1 configurations in repeated strategic games. The panel met preregistered broad-reference condition-mean criteria in three of four repeated-game cells; the sole miss was 0.011 below the lower reference bound. Variation was strongly prompt-indexed, but its share depended on uncertainty assumptions: fixed-panel symmetric-Dirichlet sensitivities produced median between-prompt shares of 63%-71% under Jeffreys alpha=0.5 and 47%-53% under alpha=1, while finite-opportunity plug-in estimates were 85%-96%. Aggregate continuation-probability contrasts were +0.083 and +0.078, with conservative simultaneous 95% intervals [-0.171, +0.330] and [-0.181, +0.330]. The treatment jointly changed the continuation process and its textual representation. A separate wording-and-position operation shifted cooperation from 0/40 to 37/40 in the bare configuration, and a label conflict also revealed representation control. The original persona-level p13 result was not prospectively family-controlled, while a post-adjudication exact gate was structurally underpowered; p13 is therefore a replication target rather than a finding. External review exposed family-error, dependence, construct, and boundary-uncertainty defects, and zero-call reanalysis changed the interpretation without rewriting the historical record. The registered marginal criteria could be passed without precisely estimating the treatment-response object. A public capsule verifies 4,916 confirmatory Phase 3-5 runs with no live model calls. The results concern one fixed model-prompt panel and do not establish human substitutability.