Designing LLM-Agents with Personalities: A Psychometric Approach

2024-10-25

arXiv.org

AI Summary

Current LLM agents lack quantifiable, psychometrically validated personality models that ensure human-comparable behavioral responses in social contexts. Method: We propose a four-stage modeling framework grounded in the Big Five Inventory (BFI), integrating semantic space alignment, scale embedding, empirical data fine-tuning, and simulated behavior generation—marking the first application of rigorous psychometric validation to LLM personality modeling. Contribution/Results: Our approach achieves high semantic consistency (0.92) in personality representation, strong correspondence with human scale responses (r = 0.87), and successfully replicates canonical associations between personality traits and risk/ethical decision-making. Critically, it establishes bidirectional interpretability between latent trait scores and observable behavioral outputs. This work advances LLMs as controllable, reproducible, and empirically verifiable tools for social science experimentation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This research introduces a novel methodology for assigning quantifiable, controllable and psychometrically validated personalities to Large Language Models-Based Agents (Agents) using the Big Five personality framework. It seeks to overcome the constraints of human subject studies, proposing Agents as an accessible tool for social science inquiry. Through a series of four studies, this research demonstrates the feasibility of assigning psychometrically valid personality traits to Agents, enabling them to replicate complex human-like behaviors. The first study establishes an understanding of personality constructs and personality tests within the semantic space of an LLM. Two subsequent studies -- using empirical and simulated data -- illustrate the process of creating Agents and validate the results by showing strong correspondence between human and Agent answers to personality tests. The final study further corroborates this correspondence by using Agents to replicate known human correlations between personality traits and decision-making behaviors in scenarios involving risk-taking and ethical dilemmas, thereby validating the effectiveness of the psychometric approach to design Agents and its applicability to social and behavioral research.

Problem

• Assigning quantifiable personalities to AI-Agents using psychometric methods
• Evaluating AI-Agent personality alignment with human responses
• Assessing AI-Agent suitability for human-substitute research applications

Innovation

• Big Five psychometric framework for AI
• LLM-based personality assignment methodology
• Prompt design with BFI-2 formats