Reason-Mediated Behavioral Models for Auditing LLM Social Simulators

2026-07-27

AI Summary

This study addresses a critical limitation in current large language models (LLMs) used for social simulation: their focus on mimicking human-like outputs while neglecting whether the generated rationales authentically reflect human decision-making logic. To bridge this gap, the authors propose an interpretable auditing framework grounded in symbolic “reason states,” which, for the first time, translates open-ended human rationales into structured signals to evaluate and guide LLM behavior in social contexts. Leveraging natural language processing, reason-state mapping, and controlled experiments that account for respondent characteristics, product categories, and concept descriptions, the research demonstrates that human-derived reason states significantly enhance purchase intention prediction accuracy. In contrast, LLM-generated rationales—though superficially plausible—largely reiterate product descriptions and fail to reconstruct genuine decision pathways, exposing the fragility of their social simulation capabilities.

Abstract

Large language models are increasingly used as social simulators, including as synthetic survey respondents. Most evaluations ask whether simulated outcomes resemble human outcomes. We argue that this is necessary but too weak: a simulator can match the final answer while using the wrong rationale-derived reason pattern. We study this problem through a 94-person sunscreen concept test in which each respondent evaluated three product concepts and wrote open-ended rationales. We map those rationales into signed reason states $Z$, where positive signs support adoption and negative signs block it. This gives a practical audit: holding respondent descriptors $D$, category context $K$, and concept treatment $X$ fixed, do human rationale-derived reasons help predict behavior $Y$, and can an LLM simulate the same reason state without seeing the human rationale or outcome? Human rationale-derived reasons substantially improve held-out prediction of purchase intent. LLM-simulated reasons are more brittle: they often sound plausible, but frequently echo the concept board rather than recover the respondent's acceptance or rejection path. The paper contributes an evaluation framework for social simulators. Reason states do not identify natural causal effects by themselves, but they provide an interpretable test of whether a simulator's stated reasons align with human evidence.

Problem

• social simulators
• reason states
• behavioral modeling
• LLM auditing
• rationale alignment

Innovation

• reason-mediated evaluation
• social simulators
• signed reason states
• LLM auditing
• behavioral rationale