Human Label Variation in Implicit Discourse Relation Recognition

2026-02-26

AI Summary

This study addresses the challenge of human annotation disagreement in implicit discourse relation recognition, demonstrating that such variability stems primarily from cognitive complexity rather than annotator bias, thereby invalidating the assumption of a single ground truth. The authors systematically compare two modeling strategies: one that predicts the full annotation distribution and another that simulates individual annotator behavior. Through controlled experiments and in-depth analysis on standard benchmarks, they find that models capturing the annotation distribution exhibit greater robustness in high-ambiguity scenarios, whereas annotator-specific models only become effective after ambiguity is reduced. These findings underscore cognitive complexity as the dominant source of annotation inconsistency and provide empirical evidence and methodological guidance for effectively modeling the diversity inherent in human annotations.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives. To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators. In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias. Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced, whereas models trained on label distributions yield more stable predictions. Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.

Problem

• Implicit Discourse Relation Recognition
• Human Label Variation
• Annotator Disagreement
• Cognitive Complexity
• Perspectivist Modeling

Innovation

• Implicit Discourse Relation Recognition
• Label Distribution Modeling
• Perspectivist Modeling
• Human Annotation Variation
• Cognitive Complexity