crowdsourcing management

This study addresses the widespread problem of incomplete reporting of annotation practices in natural language processing (NLP) research, which undermines reproducibility and quality assessment. Analyzing 1,603 papers from major NLP conferences between 2018 and 2025, the work introduces a unified taxonomy for annotation reporting that spans tasks, time, and domains, along with a minimal reporting standard. Leveraging a gold-standard dataset—Annotated-gold—curated through a combination of large language models and human adjudication, the authors construct Annotated-llm, achieving human-level inter-annotator agreement (Krippendorff’s α = 0.606) on structured information extraction. Despite gradual improvements in reporting over time, critical details—such as annotator training, linguistic competence, and compensation—remain frequently omitted. These findings advance the push toward more transparent and reliable annotation practices in NLP.

To address the tension between high annotation costs and stringent accuracy requirements, this paper proposes an active pruning method targeting minority reports—i.e., annotator responses deviating from the consensus. It is the first to jointly model image ambiguity, inter-annotator variability, and annotator fatigue as core causal factors underlying minority reports. The method introduces a dynamic redundancy reduction mechanism grounded in the probability of majority-vote bias, enabling pre-annotation identification and removal of tasks prone to erroneous labels. Experiments across multiple computer vision benchmarks demonstrate that the approach reduces annotation volume by over 60% on average, incurs only marginal label quality degradation, and saves approximately 6.6 person-days of labor—while supporting customizable accuracy–cost trade-offs. The key contributions are: (1) the first systematic causal modeling of minority report generation, and (2) a transparent, tunable framework for optimizing annotation redundancy.

Amid the growing diversity of natural language processing tasks, existing inter-annotator agreement (IAA) metrics often suffer from limited applicability and interpretability when confronted with heterogeneous task types, label imbalance, and missing data. This work systematically reviews the theoretical foundations and practical methodologies of IAA, offering the first structured integration of mainstream metrics—such as Cohen’s Kappa and Krippendorff’s Alpha—organized by task type. It clarifies their underlying assumptions and delineates their boundaries of applicability. Furthermore, the study proposes a reliability assessment strategy that combines confidence intervals with analysis of disagreement patterns. By providing a clear, principled guide for selecting IAA metrics, this research significantly enhances the transparency, reproducibility, and scientific rigor of human annotation and evaluation practices in the NLP community.

NLP data quality assessment has long relied on inter-annotator agreement, overlooking intra-annotator consistency—the temporal stability of individual annotators’ judgments. This neglect challenges the implicit “gold label as ground truth” assumption. Method: We conduct exploratory repeated annotation experiments across major NLP datasets and quantify intra-annotator agreement using Cohen’s and Fleiss’ Kappa, complemented by qualitative perceptual analysis. Contribution/Results: We demonstrate that mainstream NLP datasets routinely omit intra-annotator consistency reporting; moreover, individual annotators exhibit significant temporal variability in labeling identical texts. We identify and disentangle the dual influence of textual ambiguity and subjectivity on annotation stability. Our work establishes intra-annotator agreement as a foundational data quality metric, providing both a methodological framework and concrete guidelines for constructing more robust, reproducible NLP datasets.

This study addresses the low accuracy of large language models (LLMs) in FAIR-compliance validation of biosample metadata. We propose a structured-knowledge-guided prompting method, integrating the CEDAR template repository, domain-specific data dictionaries, and GPT-4 to construct a metadata standards-conformance verification framework—demonstrated on human lung cancer biosamples. Experimental results show that incorporating structured knowledge significantly improves field-level standards compliance from 79% to 97% (p < 0.01), providing the first empirical evidence that structured knowledge bases can overcome performance bottlenecks inherent to purely text-based LLM prompting in metadata governance. Our approach establishes a novel paradigm for automated, high-accuracy, and interpretable FAIR metadata quality control, enabling scalable, standards-aware curation of biomedical metadata.

This study addresses the limitations of large language models (LLMs) in annotating complex social science constructs—such as climate mitigation pessimism—where autonomous labeling often yields suboptimal quality. To overcome this, the authors propose AnnotateThis, a human-centered interactive annotation system that introduces an innovative “LLM grounding” paradigm, deeply integrating expert knowledge into the LLM annotation pipeline. The system enables iterative co-evolution of conceptual definitions and model refinement through human–AI collaboration, interactive visualizations, and dynamic prompt optimization, functioning effectively both with and without ground-truth labels. Empirical evaluation demonstrates that, in labeled settings, AnnotateThis achieves a 0.15 improvement in F-Measure and a 0.23 gain in accuracy, significantly outperforming existing fully automated approaches.

This work addresses the high cost and substantial cognitive load associated with structured annotation, which hinder efficient allocation of labeling efforts among heterogeneous annotators such as humans and models. The authors propose a center-theory-based task decomposition approach that identifies semantic centers to constrain the output space, formally models reasoning load, and introduces an algorithm for allocating heterogeneous annotation resources. Notably, this is the first method to integrate task decomposition with annotator capability matching. Experimental results demonstrate that the proposed approach significantly reduces cognitive burden while simultaneously improving both annotation quality and cost efficiency under a fixed budget.

This study addresses the neglect of annotators’ dynamic competence development in subjective annotation tasks by framing the annotation process itself as a skill-building mechanism, using social influence identification as a case study. Tracking the performance of 25 expert and non-expert annotators across 1,021 dialogue segments, the research integrates pre- and post-annotation comparisons, semi-structured interviews, self-assessment questionnaires, and large language model (LLM) training and evaluation based on the annotated data. Results reveal that active annotation significantly enhances annotators’ self-perceived competence and confidence, with experts exhibiting more pronounced gains. Crucially, the performance trajectory of LLMs trained on their annotations effectively mirrors this improvement, thereby validating the positive impact of the annotation process on downstream model efficacy.

This work addresses the challenge of hallucination in large language models when generating disease-related biomedical associations by proposing a verifiable and reproducible evaluation framework. The framework integrates retrieval-augmented generation (RAG), semantic matching with biomedical ontologies, majority voting across open-source large language models, and self-consistency analysis, thereby overcoming the limitations of traditional exact ontology matching and enabling multi-level semantic validation of generated content. Experimental results demonstrate that this approach significantly enhances the ability to assess the factual accuracy of associations produced by models such as ChatGPT, offering a novel paradigm for evaluating the reliability of biomedical knowledge generation.

This study addresses the bottleneck of phenotype annotation—its reliance on manual curation and limited scalability—by introducing state-of-the-art large language models (LLMs) as “agent curators” that autonomously map free-text descriptions to standardized ontology terms (UBERON, PATO, BSPO, GO) within a closed workspace. The approach integrates original literature in PDF format, curation guidelines, and semantic validation scripts, leveraging five hosted LLMs from Anthropic and OpenAI to perform end-to-end annotation. Evaluated against a Gold Standard benchmark, all LLM agents achieved performance within the inter-annotator agreement range of human curators and significantly outperformed the conventional tool Semantic CharaParser, thereby demonstrating the feasibility and superiority of LLMs for biological ontology annotation.