inter-rater reliability

Existing classifier agreement metrics (e.g., Cohen’s kappa) lack a statistical significance assessment framework, rendering their numerical values difficult to interpret objectively. Method: We propose the first general-purpose significance evaluation framework, introducing two novel indices: (i) an empirical significance index for finite samples—built upon Monte Carlo hypothesis testing and an efficient numerical algorithm—and (ii) an asymptotic significance index for classification probability distributions—characterizing statistical meaning in the large-sample limit. Contribution/Results: Our framework yields rigorous p-values and data-driven significance thresholds for any agreement metric, eliminating subjective interpretive boundaries. Empirically validated on medical evaluation and AI model compression tasks, it demonstrates robustness and practical utility, advancing the paradigm from “empirical agreement” to “statistically reliable agreement.”

Existing Cohen’s and Fleiss’ kappa statistics are restricted to single-label classification and lack robustness in realistic settings—such as comorbid psychiatric diagnoses or multi-behavior coding—where multi-label annotations, hierarchical category structures, variable numbers of annotators, and missing labels commonly occur. This paper introduces a generalized κ statistic: the first extension of Fleiss’ kappa to multi-label settings; it incorporates a category-weighting matrix and hierarchical distance metrics to capture semantic similarity among labels, and employs probabilistic expectation-based estimation to ensure robustness against missing data and variable annotator counts. Theoretically, it strictly reduces to classical Fleiss’ kappa under single-label conditions. Implemented in R and Excel, the method is validated through rigorous mathematical derivation and empirical application to psychiatric multi-diagnosis data. Its interpretable range is formally established as [−1, 1], with principled guidelines for interpretation.

This study addresses the inconsistent metric selection and inadequate reporting practices in LLM-as-judge research, which hinder reproducibility and cross-study comparison. Through a systematic analysis of agreement measures between large language models and human evaluators, the work reveals mathematical equivalences among multiple correlation coefficients—such as Pearson, Spearman, phi, and Matthews—under binary scoring, clarifies the distinct utility of Cohen’s κ, and elucidates how handling abstentions fundamentally affects evaluation outcomes. Building on these statistical insights, the paper proposes a standardized reporting checklist that encompasses rating scales, treatment of abstentions and ties, coverage, confusion matrices, and aggregation levels. This framework substantially enhances the transparency, comparability, and reproducibility of LLM-as-judge evaluations.

NLP data quality assessment has long relied on inter-annotator agreement, overlooking intra-annotator consistency—the temporal stability of individual annotators’ judgments. This neglect challenges the implicit “gold label as ground truth” assumption. Method: We conduct exploratory repeated annotation experiments across major NLP datasets and quantify intra-annotator agreement using Cohen’s and Fleiss’ Kappa, complemented by qualitative perceptual analysis. Contribution/Results: We demonstrate that mainstream NLP datasets routinely omit intra-annotator consistency reporting; moreover, individual annotators exhibit significant temporal variability in labeling identical texts. We identify and disentangle the dual influence of textual ambiguity and subjectivity on annotation stability. Our work establishes intra-annotator agreement as a foundational data quality metric, providing both a methodological framework and concrete guidelines for constructing more robust, reproducible NLP datasets.

For high-stakes decision-making domains lacking ground-truth labels—such as judicial adjudication and clinical diagnosis—this paper proposes a human-centered classifier evaluation framework. Methodologically, it introduces the *Rater Equivalence Number* (REN), the first formal metric quantifying how many human raters’ collective judgment a model’s performance equivalently matches, thereby enabling interpretable, human-aligned assessment. The framework distinguishes two utility models: *ground-truth consistency* (agreement with latent consensus) and *individual judgment matching* (fidelity to diverse human judgments), supporting value-sensitive deployment trade-offs. It integrates crowdsourced annotation modeling, statistical consistency analysis, and benchmark panel construction to jointly generate an evaluation reference standard and quantify model performance from human-labeled data. Empirical case studies and formal analysis validate its theoretical soundness and practical efficacy, establishing an actionable evaluation paradigm and deployment guidance for AI systems operating without gold-standard labels.

Amid the growing diversity of natural language processing tasks, existing inter-annotator agreement (IAA) metrics often suffer from limited applicability and interpretability when confronted with heterogeneous task types, label imbalance, and missing data. This work systematically reviews the theoretical foundations and practical methodologies of IAA, offering the first structured integration of mainstream metrics—such as Cohen’s Kappa and Krippendorff’s Alpha—organized by task type. It clarifies their underlying assumptions and delineates their boundaries of applicability. Furthermore, the study proposes a reliability assessment strategy that combines confidence intervals with analysis of disagreement patterns. By providing a clear, principled guide for selecting IAA metrics, this research significantly enhances the transparency, reproducibility, and scientific rigor of human annotation and evaluation practices in the NLP community.

This study addresses the lack of systematic statistical analysis on how modifications to scoring rubrics affect agreement between human raters and large language model (LLM)-based automated scoring systems. It presents the first quantitative investigation into the impact of various rubric design choices—including the incorporation of contextual exemplars, adjustments in complexity, and mitigation of position bias—on human–LLM scoring consistency. Drawing on data from automated essay scoring and instruction-following evaluation tasks, the authors employ statistical methods to compare outcomes under holistic versus analytic rubric configurations. Their findings reveal that integrating representative examples and reducing position bias significantly enhance inter-rater consistency, whereas highly complex rubrics and conservative aggregation strategies tend to diminish it. These results offer empirical guidance for optimizing rubric design in automated scoring systems.

This study addresses the significant bias inherent in traditional Δ coefficients and their associated parameters—category-specific contributions (α_i) and observer agreement (S_i)—when applied to small samples (n ≤ 50) or a limited number of categories (K ≤ 3), which compromises the accuracy of inter-rater reliability assessment. The authors propose, for the first time, a systematic framework of unbiased estimators for Δ, α_i, and S_i, grounded in an unbiased estimator of expected chance agreement, and rigorously analyze their bias and variance properties. The approach is further extended to scenarios involving a gold-standard rater, introducing two novel metrics: "conformity" and "predictiveness." Both theoretical analysis and simulation results demonstrate that the proposed estimators substantially outperform conventional methods under small-sample or low-category conditions, warranting their preferential use in such settings.

This study addresses the challenges of complex sample size calculations and difficult interpretation of Kappa coefficients in inter-rater agreement analysis, particularly for researchers lacking programming expertise. To this end, we developed an open-source, interactive web application based on R Shiny that integrates the core functionalities of the kappaSize and irr packages into a unified platform for the first time. The tool supports computation of Cohen’s, Fleiss’, and Light’s Kappa statistics and automatically interprets results according to the Landis & Koch benchmark scale. Offering both a graphical user interface and command-line access, it substantially lowers the technical barrier to conducting agreement analyses. Published on CRAN, this application provides an efficient, user-friendly, and reproducible solution for consistency assessment in categorical data.

Existing tools for measuring political orientation perform poorly in non-Western, data-scarce regions. This work proposes an innovative approach that treats large language models as fallible expert raters, constructing a multi-model ensemble panel guided by axially defined ideological anchors, a mechanism separating stated beliefs from observed behaviors, and explicit rules distinguishing zero scores from missing values. Rater consistency is rigorously evaluated using Krippendorff’s alpha and triple-blind assessments, achieving high inter-rater agreement (α = 0.86) across nine models. The incorporation of formal definitions significantly reduces scoring discrepancies and enhances inter-model correlations. Disagreement analysis reveals that divergences in political stance judgments stem primarily from interpretive differences rather than random error, substantially improving the method’s cross-regional applicability.