A Large-Scale Simulation on Large Language Models for Decision-Making in Political Science

2024-12-19

AI Summary

Large language models (LLMs) face fundamental challenges in simulating voter decision-making for political science—namely, data scarcity, environmental dynamism, and the complexity of human reasoning. Method: We propose a theory-driven, multi-step reasoning framework that integrates demographic, temporal, and ideological dimensions. It employs synthetic, calibrated personas grounded in real election data to enable scalable and interpretable voter behavior modeling. Contribution/Results: This work establishes the first political-science–oriented LLM-based decision simulation paradigm, incorporating bias-mitigation mechanisms and rigorously defining its applicability boundaries via cross-model robustness evaluation. Experiments demonstrate significant improvements in simulation accuracy, robust fine-grained inference under sparse real voting data, and consistent outputs across diverse LLMs—thereby providing a novel benchmark and an extensible methodology for computational political science.

Technology Category

• Humans and AI: Voting
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While LLMs have demonstrated remarkable capabilities in text generation and reasoning, their ability to simulate human decision-making -- particularly in political contexts -- remains an open question. However, modeling voter behavior presents unique challenges due to limited voter-level data, evolving political landscapes, and the complexity of human reasoning. In this study, we develop a theory-driven, multi-step reasoning framework that integrates demographic, temporal and ideological factors to simulate voter decision-making at scale. Using synthetic personas calibrated to real-world voter data, we conduct large-scale simulations of recent U.S. presidential elections. Our method significantly improves simulation accuracy while mitigating model biases. We examine its robustness by comparing performance across different LLMs. We further investigate the challenges and constraints that arise from LLM-based political simulations. Our work provides both a scalable framework for modeling political decision-making behavior and insights into the promise and limitations of using LLMs in political science research.

Problem

• Simulate human decision-making in political contexts.
• Model voter behavior using demographic, temporal, ideological factors.
• Assess LLM-based political simulation challenges and constraints.

Innovation

• Multi-step reasoning framework
• Synthetic personas calibration
• Large-scale election simulations