COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal

2026-10-05

AI Summary

This study addresses the susceptibility of language models to questionnaire wording artifacts in psychological assessment, which impedes their ability to disentangle symptom structures from genuine individual psychological signals. We propose a psychometric representational similarity analysis framework that geometrically compares natural language processing data from clinical interviews with self-reports, employing preregistered statistical validation to isolate wording effects. Our findings reveal, for the first time, that the symptom geometry generated by language models aligns predominantly with questionnaire phrasing rather than self-reported characteristics, demonstrating no evidence of wording-independent psychological structure. These results establish that current language models capture superficial textual patterns rather than deep individual psychological states, providing critical empirical evidence delineating the validity boundaries of AI-driven psychometrics.

Abstract

Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.

Problem

• psychiatric assessment
• language models
• representational similarity analysis
• questionnaire wording
• individual signal

Innovation

• Psychometric Representational Similarity Analysis
• Language Models
• COMPASS 2.0
• Symptom Geometry
• Construct Scoring