Large language models as linguistic simulators and cognitive models in human research

2024-02-06

AI Summary

Can large language models (LLMs) serve as valid substitutes for human participants in behavioral and psychological research? Method: We propose a novel paradigm—“language simulators and cognitive models”—and introduce the first five-dimensional evaluation framework. Integrating critical theoretical analysis, principles of psychological experimental design, prompt engineering standards, and model alignment mechanisms, we systematically identify six fundamental cognitive misapplication fallacies. Contribution: We establish LLMs’ theoretical role as simulation tools—not explanatory or causal instruments—and reconceptualize internal, external, construct, and statistical validity criteria accordingly. Furthermore, we provide an actionable methodological guide covering model selection, prompt design, result interpretation, and ethical review—advancing methodological rigor and paradigmatic innovation in computational social science. (149 words)

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rise of large language models (LLMs) that generate human-like text has sparked debates over their potential to replace human participants in behavioral and cognitive research. We critically evaluate this replacement perspective to appraise the fundamental utility of language models in psychology and social science. Through a five-dimension framework, characterization, representation, interpretation, implication, and utility, we identify six fallacies that undermine the replacement perspective: (1) equating token prediction with human intelligence, (2) assuming LLMs represent the average human, (3) interpreting alignment as explanation, (4) anthropomorphizing AI, (5) essentializing identities, and (6) purporting LLMs as primary tools that directly reveal the human mind. Rather than replacement, the evidence and arguments are consistent with a simulation perspective, where LLMs offer a new paradigm to simulate roles and model cognitive processes. We highlight limitations and considerations about internal, external, construct, and statistical validity, providing methodological guidelines for effective integration of LLMs into psychological research, with a focus on model selection, prompt design, interpretation, and ethical considerations. This perspective reframes the role of language models in behavioral and cognitive science, serving as linguistic simulators and cognitive models that shed light on the similarities and differences between machine intelligence and human cognition and thoughts.

Problem

• Evaluating if LLMs can replace humans in psychological research
• Identifying six fallacies in interpreting LLMs as human substitutes
• Distinguishing functional vs mechanistic equivalence between LLMs and humans

Innovation

• Critically evaluates LLM replacement fallacies
• Distinguishes functional vs mechanistic equivalence
• Proposes safeguards for responsible research practices