Subjective Logic Encodings

2025-02-17

AI Summary

In subjective tasks—such as sentiment analysis and hate speech detection—labeler disagreement reflects inherent ambiguity, yet prevailing approaches impose a “gold-standard” assumption, treating disagreement solely as noise. To address this, we propose Subjective Logic Encodings (SLEs), the first framework to integrate subjective logic theory into label modeling. SLEs explicitly represent crowdsourced annotations as annotator opinions, jointly encoding confidence, annotator reliability, and inter-annotator disagreement within a unified uncertainty-aware representation. Grounded in the Dirichlet distribution, SLEs are trained via a distribution-matching objective, generalizing multiple existing label-encoding schemes. Experiments across diverse subjective NLP benchmarks demonstrate that models trained with SLEs achieve significant improvements in calibration, robustness to annotation noise, and principled uncertainty quantification—outperforming standard label-smoothing, Bayesian, and consensus-based baselines.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Many existing approaches for learning from labeled data assume the existence of gold-standard labels. According to these approaches, inter-annotator disagreement is seen as noise to be removed, either through refinement of annotation guidelines, label adjudication, or label filtering. However, annotator disagreement can rarely be totally eradicated, especially on more subjective tasks such as sentiment analysis or hate speech detection where disagreement is natural. Therefore, a new approach to learning from labeled data, called data perspectivism, seeks to leverage inter-annotator disagreement to learn models that stay true to the inherent uncertainty of the task by treating annotations as opinions of the annotators, rather than gold-standard facts. Despite this conceptual grounding, existing methods under data perspectivism are limited to using disagreement as the sole source of annotation uncertainty. To expand the possibilities of data perspectivism, we introduce Subjective Logic Encodings (SLEs), a flexible framework for constructing classification targets that explicitly encodes annotations as opinions of the annotators. Based on Subjective Logic Theory, SLEs encode labels as Dirichlet distributions and provide principled methods for encoding and aggregating various types of annotation uncertainty -- annotator confidence, reliability, and disagreement -- into the targets. We show that SLEs are a generalization of other types of label encodings as well as how to estimate models to predict SLEs using a distribution matching objective.

Problem

• Handles inter-annotator disagreement as uncertainty
• Encodes labels using Subjective Logic Theory
• Aggregates annotator confidence, reliability, and disagreement

Innovation

• Subjective Logic Encodings framework
• Encodes annotations as opinions
• Aggregates annotation uncertainty types