Personality Structured Interview for Large Language Model Simulation in Personality Research

2025-02-17

AI Summary

Large language models (LLMs) generate personality data lacking the intrinsic heterogeneity observed in human populations, undermining ecological validity in empirical social science research. To address this, we propose the Theory-Driven Personality Structured Interview (PSI) framework, which anchors LLM generation to 357 real human interview transcripts. PSI integrates psychometrically grounded interview design, fine-grained prompt engineering, and behavioral prediction modeling to constrain and guide LLM outputs toward authentic personality diversity. Our key contribution is the first use of theory-anchored structured interviews both as generative constraints and as validation benchmarks, alongside a scalable interview paradigm and systematic evaluation protocol. Experiments demonstrate that PSI significantly enhances the predictive validity of LLM-generated responses on organizational citizenship behavior and counterproductive work behavior tasks—achieving performance statistically indistinguishable from that of human-derived data.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Understanding People, Theories, Concepts and Methods
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Although psychometrics researchers have recently explored the use of large language models (LLMs) as proxies for human participants, LLMs often fail to generate heterogeneous data with human-like diversity, which diminishes their value in advancing social science research. To address these challenges, we explored the potential of the theory-informed Personality Structured Interview (PSI) as a tool for simulating human responses in personality research. In this approach, the simulation is grounded in nuanced real-human interview transcripts that target the personality construct of interest. We have provided a growing set of 357 structured interview transcripts from a representative sample, each containing an individual's response to 32 open-ended questions carefully designed to gather theory-based personality evidence. Additionally, grounded in psychometric research, we have summarized an evaluation framework to systematically validate LLM-generated psychometric data. Results from three experiments demonstrate that well-designed structured interviews could improve human-like heterogeneity in LLM-simulated personality data and predict personality-related behavioral outcomes (i.e., organizational citizenship behaviors and counterproductive work behavior). We further discuss the role of theory-informed structured interviews in LLM-based simulation and outline a general framework for designing structured interviews to simulate human-like data for psychometric research.

Problem

• Enhance heterogeneity in LLM-simulated personality data.
• Validate LLM-generated psychometric data systematically.
• Design theory-informed interviews for human-like simulations.

Innovation

• Structured Interview for Personality Simulation
• Theory-based Personality Evidence Collection
• Framework for LLM Data Validation