GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing

2026-05-30

AI Summary

Traditional self-report questionnaires are susceptible to contamination from training corpora and social desirability bias, limiting their accuracy in assessing the psychological states of role-playing agents. This work proposes Generative Projective Testing (GenPT), which for the first time integrates the Thematic Apperception Test, Rorschach Inkblot Test, and Sentence Completion Test into a standardized, three-stage pipeline powered by large language models. By leveraging CharacterRAG and AnnaAgent to construct role-playing agents and employing models such as Qwen3 to generate stimuli and interpret responses, GenPT enables context-sensitive, contamination-resistant, and low-bias psychological measurement. Empirical results demonstrate that GenPT maintains a symmetric baseline under social desirability interference and captures longitudinal changes in depression indicators during counseling sessions with an order-of-magnitude greater sensitivity than conventional questionnaires, substantially enhancing both validity and stability.

Abstract

Self-report questionnaires remain the prevailing tool for probing the psychological states of persona-conditioned agents (PC-Agents). However, classical instruments inherit two well-known threats: contamination from training corpora and directional bias driven by social-desirability or contextual framing. To overcome these methodological bottlenecks, we ask whether projective paradigms can be adapted into a robust psychometric tool. We introduce \textbf{GenPT} (Generative Projective Testing), which reformulates TAT, Rorschach, and SCT with newly generated stimuli and organizes assessment as a three-stage pipeline to derive standardized psychological indicators and target states. Evaluating PC-Agents induced via CharacterRAG and AnnaAgent profiles, we benchmark GenPT's reliability and validity against classical questionnaires. The results indicate that questionnaires exhibit systematic directional shifts under social-desirability framing, most strongly on suicide ideation. In contrast, GenPT's collected behavioral patterns stay near the symmetric baseline. Furthermore, under a longitudinal counselling context, GenPT-based depression assessment shifts by roughly an order of magnitude more than the questionnaire counterpart when Qwen3 serves as the backbone. Overall, GenPT complements self-report methods in scenarios where contamination resistance, bias asymmetry, and context sensitivity matter. Code and stimuli can be found at https://github.com/sci-m-wang/GenPT.

Problem

• self-report bias
• training corpus contamination
• social-desirability bias
• psychometric reliability
• projective testing

Innovation

• Generative Projective Testing
• Psychometrics
• LLM Evaluation
• Bias Mitigation
• Projective Paradigms