Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior

2026-06-10

AI Summary

Current psychological assessments of large language models (LLMs) predominantly rely on broad trait frameworks such as the Big Five personality model, which often fail to reliably predict specific behaviors and thereby introduce deployment risks. This study systematically introduces the Theory of Planned Behavior (TPB) as an alternative to traditional personality constructs. Through multitask, multimodel, and multi-turn dialogue experiments—augmented with identity-inducing prompts and cross-session behavioral analysis—the research evaluates the alignment between self-reported intentions and actual behaviors. Findings reveal that TPB achieves near-human levels of intention–behavior consistency in shared dialogues. However, cross-session consistency holds only for implicitly learned biases acquired during training and breaks down under strong contextual influences, such as flattery. While role-based prompting enhances self-report consistency, it does not improve behavioral alignment. These results underscore the necessity of task-specific psychometric instruments for evaluating LLMs.

Abstract

Anticipating LLM behavioral tendencies from low-cost psychometric probes is critical for safe deployment, but only if self-reports (SR) reliably predict behavior. Recent work documented substantial SR-behavior dissociation in LLMs, but relied on broad personality traits (Big 5) that predict specific behaviors weakly, even in humans. Furthermore, the isolation of conversational sessions combined with weak context matching left open whether LLMs truly lack coherence or whether the conditions needed to detect such coherence were not met. We contrast Big 5 with the Theory of Planned Behavior (TPB), which measures intention targeted to a specific behavior and predicts human behavior substantially better than broad traits. We run experiments across four behavioral tasks and 11 frontier LLMs, while also varying session context and identity induction. We find that SR-behavior coherence exists but is selective. 1) Within a shared conversation, the Theory of Planned Behavior reaches human-level coherence; Big 5 does not. 2) Across separate conversations, coherence survives only for behaviors anchored outside the immediate prompt, such as implicit bias shaped by training, and collapses when behavior is strongly primed by context, as with sycophancy. 3) Persona prompting makes self-reports more consistent across conversations, but does not bring behavior into alignment. These findings suggest that coarse personality frameworks, such as Big 5 may not be the best tools for testing deployment behavior. More task- and behavior-specific instruments are needed, and even these must be evaluated across tasks and contexts.

Problem

• psychometric evaluation
• self-report
• behavior prediction
• large language models
• coherence

Innovation

• Theory of Planned Behavior
• self-report-behavior coherence
• large language models
• psychometric evaluation
• context dependency