Can Third-parties Read Our Emotions?

2025-04-25

AI Summary

This study investigates whether third parties—humans or large language models (LLMs)—can accurately identify private emotions and opinions expressed in author-generated text. By systematically comparing first-party (author-reported) ground truth with third-party annotations, it provides the first empirical evidence of systematic bias in existing emotion labeling practices. The methodology integrates human subject experiments, demographic特征 modeling, and LLM prompt engineering. Key contributions include: (1) demonstrating that demographic similarity between authors and human annotators significantly improves annotation accuracy; (2) showing that LLMs outperform humans overall in emotion recognition; and (3) proposing a novel prompt optimization framework incorporating author demographic information, which yields statistically significant improvements in LLM emotion classification accuracy. These findings advance the theoretical foundations and methodological toolkit for trustworthy affective computing and human-AI collaborative annotation.

Technology Category

Natural Language Processing: Ethics — Bias, Fairness, Transparency & PrivacyCognitive Modeling & Cognitive Systems: Affective ComputingHumans and AI: Emotional Intelligence

Application Category

Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labelingUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Natural Language Processing tasks that aim to infer an author's private states, e.g., emotions and opinions, from their written text, typically rely on datasets annotated by third-party annotators. However, the assumption that third-party annotators can accurately capture authors' private states remains largely unexamined. In this study, we present human subjects experiments on emotion recognition tasks that directly compare third-party annotations with first-party (author-provided) emotion labels. Our findings reveal significant limitations in third-party annotations-whether provided by human annotators or large language models (LLMs)-in faithfully representing authors' private states. However, LLMs outperform human annotators nearly across the board. We further explore methods to improve third-party annotation quality. We find that demographic similarity between first-party authors and third-party human annotators enhances annotation performance. While incorporating first-party demographic information into prompts leads to a marginal but statistically significant improvement in LLMs' performance. We introduce a framework for evaluating the limitations of third-party annotations and call for refined annotation practices to accurately represent and model authors' private states.

Problem

• Examining accuracy of third-party annotations for author emotions
• Comparing third-party and first-party emotion label reliability
• Improving annotation quality via demographic similarity and prompts

Innovation

• Compare third-party and first-party emotion annotations
• Use demographic similarity to improve annotation quality
• Evaluate limitations of third-party emotion recognition