ground-truth labeling

High annotation costs and prolonged turnaround times plague NLP development, necessitating efficient and reliable data labeling paradigms. This paper proposes an LLM-powered Human-in-the-Loop (HITL) hybrid annotation framework that systematically integrates synthetic data generation, active learning, and human-AI collaboration, augmented with built-in mechanisms for annotation quality assessment, annotator management, and cost-benefit analysis. Unlike prior work—largely theoretical or narrowly scoped—this study introduces the first deployable, plug-and-play industrial-grade annotation methodology, bridging the critical gap between methodological research and real-world engineering practice. Empirical validation across multiple production NLP projects demonstrates that the framework consistently reduces annotation costs and cycle time by 30–50%, while maintaining label quality within required thresholds.

This study addresses the “quality–diversity trade-off” in subjective annotation tasks: conventional spam annotator filtering methods treat label variation as noise, erroneously removing reliable annotators holding minority opinions and thereby distorting the true opinion distribution. The authors argue that genuine spam annotators tend to exhibit *fixed-response behavior*—repeatedly selecting identical labels—not random guessing; thus, spam behavior must be redefined accordingly. They design multiple heuristic filtering strategies and systematically evaluate them on synthetically noisy data. Results show that annotation bias remains acceptable only when ≤5% of annotators are removed; most existing methods fail to detect fixed-response spammers and frequently misclassify non-random yet reliable dissenting annotators as noise. The core contribution is the insight that label diversity itself constitutes meaningful signal—not noise—in subjective tasks, and the proposal of *response fixity*, rather than inter-annotator agreement, as a principled criterion for spam detection.

This study addresses the prevailing tendency in machine learning to mischaracterize annotation disagreement as mere noise, thereby overlooking its value as a sociotechnical signal. Through a systematic literature review and reflexive thematic analysis of 346 papers from seven top-tier conferences (2020–2025), the work uncovers the mechanisms behind the “consensus trap” and its detrimental effects on algorithmic fairness. It critiques the “noise sensor” fallacy and advocates reinterpreting disagreement as a high-fidelity signal, proposing a new annotation paradigm centered on pluralistic experiential mappings rather than a singular “ground truth.” The analysis further reveals structural inequities—including the imposition of Western norms through geographic hegemony and annotators’ compliance driven by economic precarity—highlighting the erasure of positional visibility and the role of models as mediators of bias.

Accurately measuring the proportion of policy-violating content actually encountered by users is challenged by the rarity of violations, high annotation costs, and the difficulty of conducting frequent, representative assessments. This work proposes a design-based measurement system that draws daily probability samples from user exposure streams using machine learning–assisted weighting. It enables efficient annotation through multimodal large language models, policy-guided prompting, and gold-set validation, and constructs unbiased estimators to produce prevalence metrics with confidence intervals. The system supports multidimensional post-stratification—such as by platform interface, user geography, or content age—using a single global sample, maintaining statistical unbiasedness while prioritizing high-exposure and high-risk content. This approach substantially improves monitoring efficiency, timeliness, and flexibility while significantly reducing annotation costs.

This study addresses pervasive issues in current large language model (LLM) benchmarks, including label errors, mechanical annotation biases, and ambiguous samples, which are silently propagated in downstream tasks. For the first time, item response theory (IRT) is introduced to audit LLM benchmarks. Leveraging responses from 114 models across seven multiple-choice and preference-based benchmarks, the authors develop metrics to identify samples with high probabilities of mislabeling. By integrating multi-model response analysis with supervised classification comparisons, the method successfully pinpoints the top 200 suspected mislabeled instances per benchmark at 95% precision. The findings reveal that state-of-the-art reward models agree with mislabeled samples 78% of the time—significantly higher than the 38% agreement rate of comparable models—suggesting a stronger bias toward stylistic preferences over factual accuracy and underscoring the critical impact of benchmark quality on evaluation reliability.

Widespread label noise (10–25%) in NLP benchmark datasets leads to systematic underestimation of model performance, with many purported “LLM failures” attributable to annotation errors rather than model limitations. Method: We propose LLM-as-a-judge—a framework leveraging ensemble judgments from GPT-4, Claude, and Llama, combined with consistency voting and error-sensitivity analysis to automatically detect mislabeled instances; we further apply label smoothing and confident learning for robust label recalibration. Contribution/Results: Comprehensive evaluation across the TRUE benchmark suite reveals substantial disparities in quality and efficiency among expert, crowdsourced, and LLM-generated annotations. After correction, state-of-the-art models achieve average accuracy gains of 3.2–7.8 percentage points. This work provides the first empirical evidence of systematic label-noise interference in LLM evaluation and introduces a scalable, collaborative adjudication paradigm that reframes data correction as model performance recalibration.

Traditional machine learning treats systematic variations in human annotations—such as ambiguity, interpretive disagreement, and errors—as mere noise, thereby obscuring the true sources of error. This work reframes annotation as a measurement process and introduces the first statistical framework that simultaneously accommodates both shared and individualized notions of “ground truth.” The proposed model decomposes annotation variability into four interpretable components: instance difficulty, annotator bias, contextual noise, and relational consistency. Leveraging probabilistic modeling, we estimate and validate these components on multi-annotator natural language inference data. Empirical results confirm the presence of all four sources of variation and demonstrate that the framework effectively disentangles annotator behavior, offering actionable insights for constructing higher-quality datasets.

This study addresses the limitations of relying solely on a single inter-rater reliability (IRR) threshold—such as Cohen’s kappa—as a ground-truth standard in AI-driven educational systems, which often compromises data reliability and validity. To overcome this, the work proposes an integrated framework featuring four key innovations: treating IRR as a diagnostic tool rather than a rigid inclusion criterion; transparently reporting annotator qualifications and annotation protocols; implementing bias auditing and validation mechanisms leveraging large language models; and incorporating multidimensional validity evidence through uncertainty-aware labeling, predictive criterion validation, and multimodal segment-wise annotation. Empirical validation via multimodal tutoring scenarios demonstrates that this paradigm substantially enhances both annotation quality and instructional efficacy, offering a scientifically grounded and practically actionable approach to constructing robust ground-truth data for educational AI applications.

This study addresses the inconsistency in human annotation caused by ambiguous category definitions in traditional content moderation. To resolve this, the authors propose an AI-driven constitutional annotation framework: large language models first assist humans in formulating structured, interpretable category “constitutions,” which then guide automated dual-axis labeling of intent and content safety. This approach shifts human effort from case-by-case judgments to high-level semantic definition. Evaluated on harassment, hate speech, and non-violent criminal conduct tasks, the method reduces cross-model annotation inconsistency by up to 57-fold compared to conventional paragraph-based rules and effectively exposes latent gaps in existing policy formulations.

Current benchmarks for multimodal large language models (MLLMs) commonly overlook systematic discrepancies in human annotator judgments—referred to as human label variation (HLV)—leading to an overestimation of model capabilities. This work proposes a novel evaluation protocol that explicitly models HLV within MLLM benchmarks for the first time, partitioning data into high- and low-agreement subsets based on non-aggregated human annotations. Using this framework, we conduct conditional evaluations of state-of-the-art models, including Gemma 3 and Qwen 2.5 VL, on a social media content classification task. Our findings reveal that while large models excel on high-agreement data, they often underperform medium-sized models on high-disagreement instances, demonstrating that model scale does not necessarily correlate with improved handling of subjective or ambiguous inputs. Consequently, reliance solely on consensus labels substantially inflates perceived model performance.

Manual annotation suffers from inconsistent quality and lacks systematic evaluation. Method: This paper proposes a consensus-based quality measurement framework grounded in multi-round annotation statistics, using dynamic decay of inter-annotator agreement variance as the core metric—established here as a “gold standard” for data quality. Recognizing annotators’ significant warm-up period but prohibitive cost of full-sample multiple annotation, we design a low-redundancy, high-efficiency progressive annotation protocol. The approach integrates statistical consistency analysis, variance convergence modeling, and label confidence estimation. Contribution/Results: Our paradigm substantially enhances data quality’s measurability and controllability: experiments show 3.2–7.8% accuracy gains across multiple NLP tasks and over 30% reduction in annotation redundancy.