Understanding Reliability in LLM-based Human Behavior Simulation

2026-09-16

AI Summary

研究提出ReliMap方法，通过分解和评估LLM模拟人类行为的三个层面来解决其可靠性问题，揭示了个体与群体层面上可靠性的不同影响因素。

Abstract

Large language models (LLMs) are increasingly used to simulate human survey responses and behavioral reactions, yet unreliable simulations can mislead social science conclusions. However, existing evaluations focus on end-to-end scores, leaving it unclear how different aspects of the simulation process interact to determine reliability. We propose ReliMap, which decomposes LLM-based human behavior simulation into three structured layers and evaluates reliability at both the individual level (R1) and population level (R2) across three configuration dimensions: model capacity, profile completeness, and population coverage. Through experiments across four simulation tasks and eleven LLMs, we find that all models exhibit substantial distributional bias without profile conditioning. Profile conditioning reduces this bias with diminishing returns. Larger models benefit more, and attribute informativeness matters more than quantity. Critically, R1 gains do not reliably transfer to R2--individual and population-level reliability can move in opposite directions. At the population layer, increasing coverage reduces variance but not systematic bias, with R2 stabilizing at around 50-100 individuals. These findings highlight that reliable simulation cannot be achieved by optimizing any single layer in isolation, but requires coordinated improvement across all three.

Problem

• Large language models
• human behavior simulation
• reliability
• evaluation
• distributional bias

Innovation

• ReliMap
• reliability evaluation
• profile conditioning