Measuring Value Expressions in Social Media Posts

2025-11-11

AI Summary

This study addresses the automated measurement of human values expressed in social media. Recognizing limitations in existing approaches—particularly their neglect of the subjective nature of value judgments and overreliance on inter-annotator agreement—we propose an “observer-dependent” hypothesis and develop a personalized modeling framework grounded in Schwartz’s theory of basic human values. Our method integrates large-scale human annotations (32,370 instances), semantic understanding from large language models, and an interpretivist-inspired individualized classification architecture. We introduce the first publicly available large-scale dataset annotated for value expression. Empirical results demonstrate that our model achieves significantly higher predictive consistency than conventional methods; notably, user acceptance of its outputs exceeds observed inter-human annotation agreement, thereby challenging the longstanding paradigm of “objective consensus” in value measurement.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Application Domains: Humanities & Computational Social Science
• Knowledge Representation and Reasoning: Preferences

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements

Abstract

The value alignment of sociotechnical systems has become a central debate but progress in this direction requires the measurement of the expressions of values. While the rise of large-language models offer new possible opportunities for measuring expressions of human values (e.g., humility or equality) in social media data, there remain both conceptual and practical challenges in operationalizing value expression in social media posts: what value system and operationalization is most applicable, and how do we actually measure them? In this paper, we draw on the Schwartz value system as a broadly encompassing and theoretically grounded set of basic human values, and introduce a framework for measuring Schwartz value expressions in social media posts at scale. We collect 32,370 ground truth value expression annotations from N=1,079 people on 5,211 social media posts representative of real users'feeds. We observe low levels of inter-rater agreement between people, and low agreement between human raters and LLM-based methods. Drawing on theories of interpretivism - that different people will have different subjective experiences of the same situation - we argue that value expression is (partially) in the eye of the beholder. In response, we construct a personalization architecture for classifying value expressions. We find that a system that explicitly models these differences yields predicted value expressions that people agree with more than they agree with other people. These results contribute new methods and understanding for the measurement of human values in social media data.

Problem

• Measuring human value expressions in social media posts using Schwartz value system
• Addressing low inter-rater agreement between human annotators and LLM methods
• Developing personalized framework for subjective value expression classification

Innovation

• Uses Schwartz value system for human values
• Introduces personalization architecture for classification
• Models subjective differences to improve agreement