agreement measurement

NLP data quality assessment has long relied on inter-annotator agreement, overlooking intra-annotator consistency—the temporal stability of individual annotators’ judgments. This neglect challenges the implicit “gold label as ground truth” assumption. Method: We conduct exploratory repeated annotation experiments across major NLP datasets and quantify intra-annotator agreement using Cohen’s and Fleiss’ Kappa, complemented by qualitative perceptual analysis. Contribution/Results: We demonstrate that mainstream NLP datasets routinely omit intra-annotator consistency reporting; moreover, individual annotators exhibit significant temporal variability in labeling identical texts. We identify and disentangle the dual influence of textual ambiguity and subjectivity on annotation stability. Our work establishes intra-annotator agreement as a foundational data quality metric, providing both a methodological framework and concrete guidelines for constructing more robust, reproducible NLP datasets.

Amid the growing diversity of natural language processing tasks, existing inter-annotator agreement (IAA) metrics often suffer from limited applicability and interpretability when confronted with heterogeneous task types, label imbalance, and missing data. This work systematically reviews the theoretical foundations and practical methodologies of IAA, offering the first structured integration of mainstream metrics—such as Cohen’s Kappa and Krippendorff’s Alpha—organized by task type. It clarifies their underlying assumptions and delineates their boundaries of applicability. Furthermore, the study proposes a reliability assessment strategy that combines confidence intervals with analysis of disagreement patterns. By providing a clear, principled guide for selecting IAA metrics, this research significantly enhances the transparency, reproducibility, and scientific rigor of human annotation and evaluation practices in the NLP community.

To address the rapid spread of misinformation on social media and poor model robustness caused by high annotation noise, this paper proposes a reliability-aware natural language inference (NLI) modeling paradigm. Methodologically: (1) we design EffiARA, the first annotation framework jointly modeling intra- and inter-group annotator consistency for fine-grained reliability estimation; (2) we dynamically weight training samples by annotator reliability and integrate knowledge-enhanced NLI modeling, soft-label training, and fine-tuning of Llama-3.2-1B and TwHIN-BERT-large. Contributions include: (i) releasing RUC-MCD—the first open Chinese knowledge-intensive conflict detection dataset; (ii) achieving a macro-F1 of 0.757 on RUC-MCD using Llama-3.2-1B, substantially outperforming all baselines; and (iii) empirically validating the effectiveness of reliability-weighted learning under noisy annotation conditions.

This study addresses statistical reasoning tasks at the doctoral level where ground-truth answers are unavailable. Method: We propose a novel paradigm of collaborative problem-solving and consensus-based evaluation using multiple large language models (LLMs)—GPT-4, LLaMA-3-70B, Claude-3-Opus, and Gemini-1.5-Flash—within a unified multi-model reasoning architecture. We systematically quantify inter-model agreement (via Fleiss’ Kappa) and its statistical association with answer reliability, integrating chi-square tests and confidence interval analysis to establish an interpretable, multi-model reliability assessment framework. Contribution/Results: Empirical results show that Claude-3 and GPT-4-0125-preview achieve the highest inter-model agreement, narrowest confidence intervals, and strongest alignment with the problem-generation model. Crucially, cross-model consensus significantly enhances overall answer credibility. This work establishes a reproducible, verifiable evaluation methodology for trustworthy LLM reasoning in high-uncertainty domains.

Widespread label noise (10–25%) in NLP benchmark datasets leads to systematic underestimation of model performance, with many purported “LLM failures” attributable to annotation errors rather than model limitations. Method: We propose LLM-as-a-judge—a framework leveraging ensemble judgments from GPT-4, Claude, and Llama, combined with consistency voting and error-sensitivity analysis to automatically detect mislabeled instances; we further apply label smoothing and confident learning for robust label recalibration. Contribution/Results: Comprehensive evaluation across the TRUE benchmark suite reveals substantial disparities in quality and efficiency among expert, crowdsourced, and LLM-generated annotations. After correction, state-of-the-art models achieve average accuracy gains of 3.2–7.8 percentage points. This work provides the first empirical evidence of systematic label-noise interference in LLM evaluation and introduces a scalable, collaborative adjudication paradigm that reframes data correction as model performance recalibration.

This study addresses the common yet often overlooked issue of subjective disagreement in multi-label sentiment annotation, which traditional approaches typically treat as noise and discard along with its underlying structural information. To better capture annotator uncertainty, the work proposes replacing hard majority voting with soft labels derived from vote proportions and intensity-weighted confidence, and introduces Soft Bernoulli Cross-Entropy (SoftBCE) for soft-supervised model training. Additionally, it incorporates a probabilistic alignment metric for evaluation and a data-driven diagnostic framework to analyze annotation discrepancies. Experimental results show that while hard labels yield marginally higher F1 scores, soft labels more faithfully represent the inherent uncertainty among annotators. This research establishes a novel paradigm and offers practical guidance for label aggregation, model training, and evaluation in multi-label sentiment analysis.

This work addresses the challenge of substantial annotator disagreement in subjective NLP tasks, where conventional majority voting discards valuable perspective diversity and individual annotator modeling suffers from high cost and poor generalization. The authors propose an aggregation approach based on clustering annotators by consistency and systematically evaluate four strategies—majority voting, ensemble methods, multi-label learning, and multi-task learning—across three tasks (sentiment analysis, emotion classification, and hate speech detection) and 40 multilingual datasets. Experimental results demonstrate that integrating consistency-based annotator clustering with multi-label or multi-task learning effectively preserves annotation diversity while significantly improving classification performance, outperforming both majority voting and per-annotator modeling by leveraging disagreement as informative signal rather than noise.

This work addresses the substantial and sample-dependent disagreement among human annotators when labeling inappropriate language, such as offensive or hateful content—a phenomenon that is difficult to quantify. The authors propose an “opposition index” to characterize the degree of annotator polarization and develop methods to predict this index and the associated annotation variance based on textual features. They systematically compare two approaches: direct regression to predict variance and variance estimation derived from predicted probability distributions. Both achieve moderate predictive performance. The study further reveals that samples with high opposition indices are more challenging for models to classify accurately and tend to have their toxicity systematically underestimated. This research offers a novel perspective and practical tools for understanding and modeling subjective annotation disagreement in toxic language detection.

Traditional machine learning treats systematic variations in human annotations—such as ambiguity, interpretive disagreement, and errors—as mere noise, thereby obscuring the true sources of error. This work reframes annotation as a measurement process and introduces the first statistical framework that simultaneously accommodates both shared and individualized notions of “ground truth.” The proposed model decomposes annotation variability into four interpretable components: instance difficulty, annotator bias, contextual noise, and relational consistency. Leveraging probabilistic modeling, we estimate and validate these components on multi-annotator natural language inference data. Empirical results confirm the presence of all four sources of variation and demonstrate that the framework effectively disentangles annotator behavior, offering actionable insights for constructing higher-quality datasets.