annotation adjudication

NLP data quality assessment has long relied on inter-annotator agreement, overlooking intra-annotator consistency—the temporal stability of individual annotators’ judgments. This neglect challenges the implicit “gold label as ground truth” assumption. Method: We conduct exploratory repeated annotation experiments across major NLP datasets and quantify intra-annotator agreement using Cohen’s and Fleiss’ Kappa, complemented by qualitative perceptual analysis. Contribution/Results: We demonstrate that mainstream NLP datasets routinely omit intra-annotator consistency reporting; moreover, individual annotators exhibit significant temporal variability in labeling identical texts. We identify and disentangle the dual influence of textual ambiguity and subjectivity on annotation stability. Our work establishes intra-annotator agreement as a foundational data quality metric, providing both a methodological framework and concrete guidelines for constructing more robust, reproducible NLP datasets.

Amid the growing diversity of natural language processing tasks, existing inter-annotator agreement (IAA) metrics often suffer from limited applicability and interpretability when confronted with heterogeneous task types, label imbalance, and missing data. This work systematically reviews the theoretical foundations and practical methodologies of IAA, offering the first structured integration of mainstream metrics—such as Cohen’s Kappa and Krippendorff’s Alpha—organized by task type. It clarifies their underlying assumptions and delineates their boundaries of applicability. Furthermore, the study proposes a reliability assessment strategy that combines confidence intervals with analysis of disagreement patterns. By providing a clear, principled guide for selecting IAA metrics, this research significantly enhances the transparency, reproducibility, and scientific rigor of human annotation and evaluation practices in the NLP community.

This work investigates whether large language models (LLMs) can effectively model human annotation disagreement—a critical signal of task subjectivity and instance ambiguity. Current evaluation paradigms predominantly assess accuracy against majority-voted labels, neglecting models’ capacity to capture annotation uncertainty. To address this gap, we propose the first systematic evaluation framework for disagreement prediction grounded in single-annotator labels, integrated with RLVR-style reasoning to quantify LLMs’ fidelity to empirical annotation distributions. Our experiments reveal three key findings: (1) mainstream LLMs exhibit poor calibration in predicting human disagreement; (2) majority-label accuracy substantially obscures this limitation; and (3) incorporating reinforcement learning–based reasoning degrades disagreement prediction performance, exposing a misalignment between standard optimization objectives and uncertainty modeling. We publicly release our code and datasets to advance more holistic, human-centered LLM evaluation.

This work addresses the challenge of annotator disagreement in subjective and ambiguous natural language processing tasks—such as toxicity detection and stance analysis—where divergent perspectives are often dismissed as noise rather than meaningful signals. The authors propose a domain-agnostic taxonomy of annotation disagreement alongside a unified modeling framework that explicitly captures structural relationships among annotators and supports multi-target prediction. By introducing disagreement-aware evaluation metrics, the study advocates a paradigm shift from consensus-based learning toward perspectivist modeling, offering a normative lens for fairness assessment. The paper systematically integrates existing disagreement-aware methodologies, clarifies the trajectory of this evolving paradigm, and outlines promising future directions, including the incorporation of multi-source variability and the development of interpretable disagreement frameworks.

Widespread label noise (10–25%) in NLP benchmark datasets leads to systematic underestimation of model performance, with many purported “LLM failures” attributable to annotation errors rather than model limitations. Method: We propose LLM-as-a-judge—a framework leveraging ensemble judgments from GPT-4, Claude, and Llama, combined with consistency voting and error-sensitivity analysis to automatically detect mislabeled instances; we further apply label smoothing and confident learning for robust label recalibration. Contribution/Results: Comprehensive evaluation across the TRUE benchmark suite reveals substantial disparities in quality and efficiency among expert, crowdsourced, and LLM-generated annotations. After correction, state-of-the-art models achieve average accuracy gains of 3.2–7.8 percentage points. This work provides the first empirical evidence of systematic label-noise interference in LLM evaluation and introduces a scalable, collaborative adjudication paradigm that reframes data correction as model performance recalibration.

Existing metrics for factuality and faithfulness struggle to evaluate how language models handle documents containing both supporting and contradictory evidence. This work proposes ConflictScore, the first formal and quantitative framework for assessing a model’s ability to recognize and articulate conflicting evidence. It decomposes model responses into atomic claims, fine-grained labels their relationships with all source documents, and introduces two complementary metrics: CS-C (Conflict Sensitivity) and CS-R (Response Reasonableness). Built upon this framework, the ConflictBench benchmark encompasses diverse conflict types. Experiments demonstrate that ConflictScore effectively identifies overconfident claims across domains and serves as a feedback signal that significantly improves model truthfulness on TruthfulQA.

This work addresses the pervasive disagreements in AI safety labeling, which often stem from ambiguities indistinguishable as operational errors, policy vagueness, or divergent value systems. To disentangle these sources, the paper introduces Annotator Policy Models (APMs)—interpretable models that learn and visualize individual annotators’ internal interpretations of safety policies solely from their labeling behavior, without requiring additional elicitation. This approach enables direct comparison of annotators’ decision logics and, for the first time, reveals their implicit policy understandings without extra annotations. By integrating interpretable machine learning, counterfactual prediction, and controlled experimentation, APMs effectively identify both policy ambiguities and systematic value-based differences. Empirically, APMs achieve over 80% accuracy in modeling annotator behavior, providing a data-driven foundation for developing more transparent and inclusive AI safety guidelines.

This work addresses the challenge of annotation disagreement in subjective NLP tasks, where ambiguity in labeling criteria or overlapping category boundaries often leads to inconsistent judgments. The authors propose a pattern-level diagnostic framework that introduces an interpretable, criterion-level auditing mechanism prior to label aggregation. By collecting fine-grained evaluations from multiple annotators on individual labeling guidelines, the method systematically identifies two failure modes: unstable annotation standards and systematic category overlap. This approach provides the first structured attribution of disagreement early in the annotation pipeline, offering empirical grounding for refining annotation schemes. Evaluated on a commercial document task involving persuasion-value extraction, the analysis reveals that disagreements concentrate around a few unstable criteria, with nearly half of the sentences activating multiple categories. The diagnostic outcomes show strong alignment with domain expert assessments and effectively inform revisions to the annotation guidelines.

Current evaluation methods struggle to uncover substantial disagreements among large language models (LLMs) in public opinion classification, potentially misleading policy decisions. This work proposes an interpretability-focused auditing framework that treats inter-model disagreement as a signal of semantic complexity, directing human review toward genuinely ambiguous opinions. Through multi-model comparisons, expert-defined scoring rules, and a two-stage annotation experiment, the study finds that thematic disagreements across models significantly outweigh variations caused by prompt perturbations within a single model. While expert rules mitigate superficial discrepancies, they fail to resolve deeper cognitive divergences. Moreover, human annotators frequently introduce novel interpretive frameworks absent from model outputs. Moving beyond conventional accuracy metrics, this paradigm highlights the diagnostic value of disagreement in interpretive coding for nuanced opinion analysis.

This study addresses the common yet often overlooked issue of subjective disagreement in multi-label sentiment annotation, which traditional approaches typically treat as noise and discard along with its underlying structural information. To better capture annotator uncertainty, the work proposes replacing hard majority voting with soft labels derived from vote proportions and intensity-weighted confidence, and introduces Soft Bernoulli Cross-Entropy (SoftBCE) for soft-supervised model training. Additionally, it incorporates a probabilistic alignment metric for evaluation and a data-driven diagnostic framework to analyze annotation discrepancies. Experimental results show that while hard labels yield marginally higher F1 scores, soft labels more faithfully represent the inherent uncertainty among annotators. This research establishes a novel paradigm and offers practical guidance for label aggregation, model training, and evaluation in multi-label sentiment analysis.