🤖 AI Summary
This study addresses the challenge of reliably and scalably measuring expressions of human basic values—grounded in Schwartz’s theory—in unstructured social media text. To this end, the authors propose a theory-constrained calibration mechanism for large language model (LLM) annotations, integrating multilingual prompt engineering, iterative error analysis, and expert-validated rules to generate soft labels that preserve semantic structure and uncertainty. These calibrated labels are then transferred to an encoder-based model to enable large-scale value prediction. The approach significantly reduces value misclassification rates, enhances agreement with expert annotations, and improves alignment with the theoretical value structure, thereby establishing a scalable framework for value detection that maintains strong fidelity to Schwartz’s theory.
📝 Abstract
Measuring subjective constructs in naturally occurring social media text requires annotation procedures that are theoretically grounded, empirically validated, and transferable to an encoder model for scalable prediction. Using non-English social media posts annotated according to Schwartz's theory of basic human values, we investigate how different LLMs, prompts, and instruction languages operationalize the expression of values in text. We argue that although texts may permit multiple plausible interpretations, theory-based value definitions can constrain interpretations and reduce spurious value attributions. Beyond precision, recall, and F1, we evaluate structural alignment between values, error structure, confidence-ambiguity relations, and annotation stability. We show that different LLMs produce different value interpretations. Iterative prompt calibration through error analysis reduces misattributions and improves alignment with expert annotations. We also derive targeted expert verification rules from recurrent error structures and use them during corpus annotation. Finally, we show that LLM annotations can be transferred to an encoder model through soft-label training, retaining theory-based value interpretations and information about uncertainty in value expression.