manage annotation pipelines

This study addresses the widespread problem of incomplete reporting of annotation practices in natural language processing (NLP) research, which undermines reproducibility and quality assessment. Analyzing 1,603 papers from major NLP conferences between 2018 and 2025, the work introduces a unified taxonomy for annotation reporting that spans tasks, time, and domains, along with a minimal reporting standard. Leveraging a gold-standard dataset—Annotated-gold—curated through a combination of large language models and human adjudication, the authors construct Annotated-llm, achieving human-level inter-annotator agreement (Krippendorff’s α = 0.606) on structured information extraction. Despite gradual improvements in reporting over time, critical details—such as annotator training, linguistic competence, and compensation—remain frequently omitted. These findings advance the push toward more transparent and reliable annotation practices in NLP.

To address the high labor cost and low efficiency of manual annotation in large-scale content labeling, this paper focuses on code documentation analysis and proposes MCHR, a semi-automated annotation framework. Methodologically, MCHR introduces (1) a novel multi-large-language-model (LLM) structured consensus mechanism that enhances robustness of automated labeling through collaborative reasoning and consensus aggregation; and (2) a difficulty-aware human review triggering protocol that dynamically invokes human intervention based on task complexity. The framework supports diverse annotation tasks—from binary classification to open-set labeling—and employs open-set evaluation to rigorously assess generalization capability. Experimental results demonstrate that MCHR achieves stable accuracy of 85.5%–98%, while reducing annotation time by 32%–100% compared to fully manual labeling. This significantly improves the scalability and practicality of annotation systems without compromising reliability.

This study addresses the overreliance on inter-annotator agreement in current data annotation practices, which often overlooks annotation’s capacity to capture conceptual validity as a measurement act. Treating annotation as a measurement process, the work identifies five root causes of annotation issues—errors, ambiguity, impossibility, subjectivity, and annotator identity—and develops a measurement theory–based framework for diagnosing and improving annotation quality. Drawing on a synthesis of 132 literature sources and 10 semi-structured interviews, the research systematically defines target constructs, designs annotation instruments, implements labeling procedures, and evaluates both reliability and validity. The resulting framework equips annotation teams with evaluation methods that transcend mere agreement metrics, thereby substantially strengthening the foundational quality of AI training data.

Amid the growing diversity of natural language processing tasks, existing inter-annotator agreement (IAA) metrics often suffer from limited applicability and interpretability when confronted with heterogeneous task types, label imbalance, and missing data. This work systematically reviews the theoretical foundations and practical methodologies of IAA, offering the first structured integration of mainstream metrics—such as Cohen’s Kappa and Krippendorff’s Alpha—organized by task type. It clarifies their underlying assumptions and delineates their boundaries of applicability. Furthermore, the study proposes a reliability assessment strategy that combines confidence intervals with analysis of disagreement patterns. By providing a clear, principled guide for selecting IAA metrics, this research significantly enhances the transparency, reproducibility, and scientific rigor of human annotation and evaluation practices in the NLP community.

NLP data quality assessment has long relied on inter-annotator agreement, overlooking intra-annotator consistency—the temporal stability of individual annotators’ judgments. This neglect challenges the implicit “gold label as ground truth” assumption. Method: We conduct exploratory repeated annotation experiments across major NLP datasets and quantify intra-annotator agreement using Cohen’s and Fleiss’ Kappa, complemented by qualitative perceptual analysis. Contribution/Results: We demonstrate that mainstream NLP datasets routinely omit intra-annotator consistency reporting; moreover, individual annotators exhibit significant temporal variability in labeling identical texts. We identify and disentangle the dual influence of textual ambiguity and subjectivity on annotation stability. Our work establishes intra-annotator agreement as a foundational data quality metric, providing both a methodological framework and concrete guidelines for constructing more robust, reproducible NLP datasets.

This study addresses the inconsistency in human annotation caused by ambiguous category definitions in traditional content moderation. To resolve this, the authors propose an AI-driven constitutional annotation framework: large language models first assist humans in formulating structured, interpretable category “constitutions,” which then guide automated dual-axis labeling of intent and content safety. This approach shifts human effort from case-by-case judgments to high-level semantic definition. Evaluated on harassment, hate speech, and non-violent criminal conduct tasks, the method reduces cross-model annotation inconsistency by up to 57-fold compared to conventional paragraph-based rules and effectively exposes latent gaps in existing policy formulations.

This study addresses the widespread neglect in machine learning research of when validation occurs during data annotation—a critical factor influencing both label quality and cost—despite overreliance on post-hoc quality control. Drawing inspiration from the “shift-left” principle in software engineering, this work proposes a tripartite classification of quality checkpoints across early, intermediate, and late stages of the annotation pipeline and introduces a parameterized error propagation model that, for the first time, treats validation timing as a quantifiable design variable. Through error propagation modeling, process decomposition, and literature analysis, the authors find that only 4% of recent studies report validation timing. Their analysis demonstrates that early-stage quality checks can reduce error correction costs by up to two orders of magnitude. The paper calls for standardized reporting of timing configurations, platform support for tunable timing parameters, and empirical studies on stage-specific detection rates.

This work addresses the high cost and substantial cognitive load associated with structured annotation, which hinder efficient allocation of labeling efforts among heterogeneous annotators such as humans and models. The authors propose a center-theory-based task decomposition approach that identifies semantic centers to constrain the output space, formally models reasoning load, and introduces an algorithm for allocating heterogeneous annotation resources. Notably, this is the first method to integrate task decomposition with annotator capability matching. Experimental results demonstrate that the proposed approach significantly reduces cognitive burden while simultaneously improving both annotation quality and cost efficiency under a fixed budget.

Behavioral profiling (BP) annotation is challenging to automate due to its multidimensional, multilingual nature, and conventional task-level evaluation obscures underlying skill heterogeneity. This work proposes a novel “skill feasibility” paradigm, decomposing BP annotation into 14 operationalizable annotation skills and implementing a schema-guided, skill-document-driven pipeline. Evaluation over a 300-instance validation set—through two rounds of testing involving human annotators and large language models (GPT-5.4 and three open-source models)—reveals a “shared categorization, independent execution” pattern: humans and GPT exhibit high agreement at the skill level but diverge in instance-level execution. The study identifies five directly feasible skills, four recoverable via relabeling, and five structurally undefined. GPT-5.4 demonstrates reliable performance on feasible skills (accuracy = 0.678, κ = 0.665, weighted F1 = 0.695), whereas open-source models primarily fail in translating schemas into executable skills.