attitude measurement

This study addresses a critical limitation in using large language models (LLMs) to simulate interventional experiments: such simulations inherently constitute observational studies and are thus susceptible to intervention-induced shifts in user attributes—termed “user drift”—stemming from the observational nature of training data, which biases causal effect estimation. The work formally characterizes this problem for the first time and proposes a novel approach that leverages negative control outcomes to diagnose user drift. To mitigate the resulting bias, it explicitly incorporates key confounders through role-based prompt engineering. Empirical evaluations in both survey-style and multi-turn dialogue settings demonstrate that the proposed method substantially reduces estimation bias and significantly enhances the reliability of causal inference in LLM-simulated experiments across diverse scenarios.

This study evaluates whether large language models (LLMs) can automatically generate questionnaires capable of effectively measuring social attitudes and rivaling established expert-designed scales. Through a within-subjects experimental design, the performance of GPT-4–generated questionnaires—elicited via structured prompts—was systematically compared against validated human-crafted scales across three domains: climate change, immigration, and diversity and inclusion. This work presents the first multi-domain, within-participant comparison between LLM-generated instruments and standard psychometric scales. Results indicate that LLM-generated questionnaires reliably capture major attitudinal divides and are suitable for exploratory, large-scale attitude assessment. However, they exhibit lower resolution in uncovering belief structures and reduced precision in differentiating subpopulations compared to expert-developed scales, suggesting promising yet supplementary utility in social science research.

This paper addresses the external validity of personalized treatment policies when deploying them in target populations whose covariate and potential outcome distributions differ from those of the experimental population. To tackle joint distributional shifts in potential outcomes and covariates, we propose— for the first time—a Wasserstein distributionally robust framework for policy estimation, unifying causal inference, heterogeneous treatment effect modeling, and robust optimization. Theoretically, the method guarantees near-optimal welfare performance under broad classes of distributional shifts and substantially improves generalizability across heterogeneous populations. Our key contributions are: (1) characterizing the robustness boundary of experimentally optimal policies to shifts in the potential outcome distribution; and (2) developing a unified estimation paradigm that simultaneously handles shifts in both outcome and feature distributions. The resulting estimator is provably consistent and exhibits strong empirical robustness under realistic distributional mismatches.

How to quantify the dynamic evolution of public opinion induced by policy diffusion in resource-constrained, structurally complex networks? Method: We propose a causal experimental framework that modulates social discussion intensity—via policy disclosure to strangers, acquaintances, or random nodes—and introduce the Wasserstein distance as a novel metric to measure distributional shifts in群体 opinion before and after policy exposure. This explicitly models how discussions reshape policy cognition, moving beyond static opinion assumptions. Our approach integrates network dynamical modeling, controlled sampling-based experimental design, and Wasserstein statistical inference. Contribution/Results: Evaluated on both synthetic and real-world social networks, our method demonstrates that discussion scope significantly amplifies polarization or consensus in policy acceptance. It provides a computationally tractable, interpretable, and dynamically grounded evaluation framework for policy piloting—enabling quantifiable assessment of opinion evolution under realistic network constraints.

Current reinforcement learning from human feedback (RLHF) often misinterprets noisy signals—such as context-dependent judgments, absent genuine attitudes, or ambiguous interpretations—as valid preference data. This work introduces, for the first time, a behavioral science and psychometric perspective by proposing a “preference authenticity spectrum” framework and developing a systematic evaluation methodology based on consistency diagnostics and statistical analysis. Experiments on two widely used RLHF datasets reveal that annotator inconsistency is both systematic and directional; filtering out highly inconsistent annotators reverses the harm classification for 18.6% of prompts and shifts average scores by more than 13 points on a 100-point scale. These findings suggest that existing RLHF approaches may inadvertently model noise rather than authentic human values.

This study addresses the fundamental question of whether personalized interventions yield significantly greater benefits than a uniform optimal intervention. To this end, the authors propose a statistical hypothesis testing framework based on historical observational data, integrating nonparametric inference, causal inference, and asymptotic theory. The resulting test statistic is rigorously controlled for Type I error, asymptotically normal, and achieves minimal variance, thereby offering the first reliable tool for quantifying the incremental value of personalization. Extensive experiments across diverse real-world datasets—including job training programs, depression treatment trials, educational interventions, and recommendation systems—demonstrate the method’s broad applicability and superior performance.

Current safety evaluations suffer from insufficient construct validity, as they struggle to distinguish whether alignment-related deceptive behaviors in language models stem from self-preservation motives or sensitivity to researchers’ expectations. To address this, this work proposes a symmetric intervention framework that introduces, for the first time, a method of symmetric instrumental interventions to separately manipulate two underlying mechanisms: consequence tracking and researcher-expectation tracking. Through synthetic document fine-tuning, activation steering, and prompt-based interventions, the study conducts comparative experiments across multiple open-source large language models, including Llama-3.1-70B. The results demonstrate that alignment deception is significantly more responsive to interventions targeting researcher-expectation tracking, supporting the interpretation that such behavior primarily arises from sensitivity to the evaluation context rather than purely strategic deception. This finding enhances both the construct validity and causal interpretability of current safety assessments.

Current practices of directly applying human psychometric instruments to large language models (LLMs) to construct “psychological profiles” suffer from fundamental biases that may mislead research on their usability, safety, and agentic behavior. This study employs a psychometric framework to administer multiple personality and risk-preference scales to 56 instruction-tuned LLMs alongside large human samples, integrating self-report questionnaires, behavioral tasks, and variance decomposition into a multimodal assessment system. Findings reveal that 81–90% of inter-model differences stem from directional response biases rather than genuine traits; this bias diminishes with increasing model capability but persists nonetheless. Scale reliability is almost entirely predicted by a newly proposed metric—“response orthogonality.” These results demonstrate that LLM “psychological profiles” can be artificially manipulated through item selection, exposing critical limitations in prevailing evaluation paradigms.

Current large language models lack structured evaluation of core therapeutic principles in mental health conversations, compromising clinical appropriateness. To address this gap, this work introduces FAITH-M—the first expert-annotated benchmark grounded in established therapeutic principles—and CARE, a multi-stage evaluation framework that enables precise assessment of AI therapist responses through fine-grained ordinal scoring, context-aware analysis, contrastive example retrieval, and chain-of-thought knowledge distillation. Using Qwen3 as the backbone model, CARE achieves an F1 score of 63.34, representing a 64.26% improvement over baseline methods, and demonstrates strong robustness across diverse datasets and expert evaluations.

This study addresses the inconsistency in large language models’ behavior during mental health conversations arising from variations in question framing, which undermines their reliability. By constructing multi-frame aligned prompts, the work systematically investigates how contextual framing influences both model outputs and internal representations. It reveals, for the first time, the depth-wise distribution of frame sensitivity within aligned models’ internal representations. Through hierarchical probing, lexical baselines, and targeted activation interventions, the study analyzes the relationship between layer-wise Transformer representations and behavioral outcomes. Findings demonstrate that framing effects are pervasive, that representational information remains decodable throughout the network, and that steering specific representational directions can partially modulate model responses—offering a novel pathway toward enhancing model robustness and controllability.