Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors

2024-10-17

arXiv.org

AI Summary

This work identifies label aggregation as a critical confounding factor degrading large language models’ (LLMs) in-context learning (ICL) performance on subjective tasks—e.g., sentiment and moral judgment. Through posterior bias analysis quantified via prior-aware metrics, annotator-level modeling, and controlled ICL experiments, we demonstrate that aggregation substantially distorts the LLM’s posterior predictive distribution: it introduces noise while inadvertently amplifying perspectives from annotators whose judgments better align with the model’s prior. To our knowledge, this is the first systematic study advocating for annotator-level modeling over conventional label aggregation. We further show that aggregation accounts for only a portion of the observed performance gap; residual subjectivity remains unexplained. Our core contribution is the formal characterization of aggregation-induced bias mechanisms, providing both theoretical grounding and methodological guidance for developing more robust and equitable ICL frameworks for subjective tasks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

In-context Learning (ICL) has become the primary method for performing natural language tasks with Large Language Models (LLMs). The knowledge acquired during pre-training is crucial for this few-shot capability, providing the model with task priors. However, recent studies have shown that ICL predominantly relies on retrieving task priors rather than"learning"to perform tasks. This limitation is particularly evident in complex subjective domains such as emotion and morality, where priors significantly influence posterior predictions. In this work, we examine whether this is the result of the aggregation used in corresponding datasets, where trying to combine low-agreement, disparate annotations might lead to annotation artifacts that create detrimental noise in the prompt. Moreover, we evaluate the posterior bias towards certain annotators by grounding our study in appropriate, quantitative measures of LLM priors. Our results indicate that aggregation is a confounding factor in the modeling of subjective tasks, and advocate focusing on modeling individuals instead. However, aggregation does not explain the entire gap between ICL and the state of the art, meaning other factors in such tasks also account for the observed phenomena. Finally, by rigorously studying annotator-level labels, we find that it is possible for minority annotators to both better align with LLMs and have their perspectives further amplified.

Problem

• Aggregation artifacts in subjective tasks
• Posterior bias towards certain annotators
• Impact of annotation artifacts on LLM predictions

Innovation

• Examines aggregation artifacts in datasets
• Focuses on individual-level modeling
• Studies annotator-level label impacts