annotation guideline design

This work addresses the challenges of information retrieval (IR) for low-resource languages, where high-quality annotated data is scarce and automatically generated labels often suffer from reliability issues and biases. The authors propose BETA-Labeling, a novel framework that systematically evaluates the effectiveness of large language model (LLM)-assisted annotation in low-resource IR. By integrating multi-model collaborative labeling, context alignment, consistency verification, and majority voting—augmented with human evaluation—they construct the first high-quality Bengali IR dataset. The study also investigates the feasibility of reusing single-hop machine-translated data from other low-resource languages, revealing performance risks in cross-lingual transfer due to inconsistent semantic preservation and language-dependent biases. Experimental results demonstrate that the proposed approach substantially improves annotation quality, while the efficacy of cross-lingual data reuse is shown to be highly dependent on the linguistic characteristics of the language pair involved.

This study addresses the widespread problem of incomplete reporting of annotation practices in natural language processing (NLP) research, which undermines reproducibility and quality assessment. Analyzing 1,603 papers from major NLP conferences between 2018 and 2025, the work introduces a unified taxonomy for annotation reporting that spans tasks, time, and domains, along with a minimal reporting standard. Leveraging a gold-standard dataset—Annotated-gold—curated through a combination of large language models and human adjudication, the authors construct Annotated-llm, achieving human-level inter-annotator agreement (Krippendorff’s α = 0.606) on structured information extraction. Despite gradual improvements in reporting over time, critical details—such as annotator training, linguistic competence, and compensation—remain frequently omitted. These findings advance the push toward more transparent and reliable annotation practices in NLP.

Widespread label noise (10–25%) in NLP benchmark datasets leads to systematic underestimation of model performance, with many purported “LLM failures” attributable to annotation errors rather than model limitations. Method: We propose LLM-as-a-judge—a framework leveraging ensemble judgments from GPT-4, Claude, and Llama, combined with consistency voting and error-sensitivity analysis to automatically detect mislabeled instances; we further apply label smoothing and confident learning for robust label recalibration. Contribution/Results: Comprehensive evaluation across the TRUE benchmark suite reveals substantial disparities in quality and efficiency among expert, crowdsourced, and LLM-generated annotations. After correction, state-of-the-art models achieve average accuracy gains of 3.2–7.8 percentage points. This work provides the first empirical evidence of systematic label-noise interference in LLM evaluation and introduces a scalable, collaborative adjudication paradigm that reframes data correction as model performance recalibration.

NLP data quality assessment has long relied on inter-annotator agreement, overlooking intra-annotator consistency—the temporal stability of individual annotators’ judgments. This neglect challenges the implicit “gold label as ground truth” assumption. Method: We conduct exploratory repeated annotation experiments across major NLP datasets and quantify intra-annotator agreement using Cohen’s and Fleiss’ Kappa, complemented by qualitative perceptual analysis. Contribution/Results: We demonstrate that mainstream NLP datasets routinely omit intra-annotator consistency reporting; moreover, individual annotators exhibit significant temporal variability in labeling identical texts. We identify and disentangle the dual influence of textual ambiguity and subjectivity on annotation stability. Our work establishes intra-annotator agreement as a foundational data quality metric, providing both a methodological framework and concrete guidelines for constructing more robust, reproducible NLP datasets.

This work systematically evaluates the efficacy and limitations of large language models (LLMs) as annotators for subjective tasks. We survey 12 prior studies and empirically compare opinion distribution alignment between GPT-series models and human annotators across four subjective datasets—introducing, for the first time, a novel evaluation paradigm centered on *perspective diversity alignment*. Results reveal substantial distributional biases in LLMs, including underrepresentation of minority viewpoints, prompt sensitivity, English-language bias, and embedded societal prejudices; most existing annotation methods overlook such distributional discrepancies, and only a few strategies effectively capture opinion diversity. Our findings expose critical reliability risks in deploying LLMs for subjective annotation and establish a reproducible statistical framework—comprising quantitative metrics and methodological guidelines—for assessing annotation quality in subjective NLP tasks.

Traditional machine learning treats systematic variations in human annotations—such as ambiguity, interpretive disagreement, and errors—as mere noise, thereby obscuring the true sources of error. This work reframes annotation as a measurement process and introduces the first statistical framework that simultaneously accommodates both shared and individualized notions of “ground truth.” The proposed model decomposes annotation variability into four interpretable components: instance difficulty, annotator bias, contextual noise, and relational consistency. Leveraging probabilistic modeling, we estimate and validate these components on multi-annotator natural language inference data. Empirical results confirm the presence of all four sources of variation and demonstrate that the framework effectively disentangles annotator behavior, offering actionable insights for constructing higher-quality datasets.

This study systematically evaluates the generalization of commonsense knowledge in large language models across multilingual and multicultural contexts, with a particular focus on low-resource languages and underrepresented cultures. Building upon a human-curated extension of the BLEnD benchmark encompassing over 30 language–culture pairs, the evaluation features two tracks—short-answer and multiple-choice—and strictly enforces a zero-shot setting, prohibiting any training or fine-tuning on the benchmark data while allowing participation from any NLP system. As the first large-scale, purely evaluative benchmark for cross-cultural commonsense reasoning, the initiative attracted registrations from over 140 teams, with 62 submitting results. Analysis reveals that state-of-the-art approaches perform substantially worse on low-resource languages, highlighting critical challenges in cultural alignment and cross-cultural commonsense transfer.

This study addresses the inconsistency in human annotation caused by ambiguous category definitions in traditional content moderation. To resolve this, the authors propose an AI-driven constitutional annotation framework: large language models first assist humans in formulating structured, interpretable category “constitutions,” which then guide automated dual-axis labeling of intent and content safety. This approach shifts human effort from case-by-case judgments to high-level semantic definition. Evaluated on harassment, hate speech, and non-violent criminal conduct tasks, the method reduces cross-model annotation inconsistency by up to 57-fold compared to conventional paragraph-based rules and effectively exposes latent gaps in existing policy formulations.

This study addresses the lack of systematic understanding regarding how annotator characteristics and textual linguistic properties jointly influence annotation variability in harmful language detection. Integrating sociolinguistic features of annotators—including demographic attributes and attitudinal measures—with computational linguistic metrics of text, the authors conduct large-scale statistical modeling and joint analyses across four harmful language datasets. They uncover significant interaction effects between annotator and text features, demonstrating that lexical cues and annotator attitudes exert strong, interdependent influences on labeling outcomes. Notably, these interaction patterns vary substantially across datasets. These findings challenge prevailing practices that overlook such complexities and underscore the necessity of explicitly accounting for annotator–text interactions when developing and generalizing harmful language detection models.