LLM Social Simulations Are a Promising Research Method

2025-04-03

AI Summary

Current large language models (LLMs) show nascent promise as “artificial subjects” for social simulation, yet suffer from weak empirical foundations, methodological unreliability, and low scholarly adoption. To address this, we propose the first systematic LLM-based social simulation framework explicitly designed for verifiability and empirical usability. Methodologically, it (i) identifies and jointly mitigates five core challenges; (ii) integrates empirical behavioral comparison benchmarks with multimodal enhancement strategies—novel in this domain; and (iii) unifies prompt engineering, domain-specific fine-tuning, and human behavioral modeling into an iterative, closed-loop evaluation architecture. Experimental results demonstrate that the framework enables low-cost pilot studies and rapid prototyping across psychology and economics, while establishing a dynamic assessment and conceptual modeling paradigm aligned with AI evolution. This advances LLM-driven social simulation from theoretical speculation toward a rigorous, empirically grounded scientific tool.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Accurate and verifiable large language model (LLM) simulations of human research subjects promise an accessible data source for understanding human behavior and training new AI systems. However, results to date have been limited, and few social scientists have adopted these methods. In this position paper, we argue that the promise of LLM social simulations can be achieved by addressing five tractable challenges. We ground our argument in a literature survey of empirical comparisons between LLMs and human research subjects, commentaries on the topic, and related work. We identify promising directions with prompting, fine-tuning, and complementary methods. We believe that LLM social simulations can already be used for exploratory research, such as pilot experiments for psychology, economics, sociology, and marketing. More widespread use may soon be possible with rapidly advancing LLM capabilities, and researchers should prioritize developing conceptual models and evaluations that can be iteratively deployed and refined at pace with ongoing AI advances.

Problem

• LLM simulations need accuracy verification for human behavior studies
• Current LLM social simulations lack widespread adoption by social scientists
• Five tractable challenges must be addressed to improve LLM simulations

Innovation

• LLM simulations for human behavior research
• Prompting and fine-tuning for better accuracy
• Iterative models aligned with AI advances