Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

2026-05-25

AI Summary

This study addresses the inconsistent metric selection and inadequate reporting practices in LLM-as-judge research, which hinder reproducibility and cross-study comparison. Through a systematic analysis of agreement measures between large language models and human evaluators, the work reveals mathematical equivalences among multiple correlation coefficients—such as Pearson, Spearman, phi, and Matthews—under binary scoring, clarifies the distinct utility of Cohen’s κ, and elucidates how handling abstentions fundamentally affects evaluation outcomes. Building on these statistical insights, the paper proposes a standardized reporting checklist that encompasses rating scales, treatment of abstentions and ties, coverage, confusion matrices, and aggregation levels. This framework substantially enhances the transparency, comparability, and reproducibility of LLM-as-judge evaluations.

Abstract

Validating an LLM judge against human annotations usually means reporting several agreement statistics: accuracy, precision, recall, $F_1$, Cohen's $κ$, and one or more rank correlations. A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling, and those choices rarely stated. For binary criteria -- the common case in rubric-based evaluation, where each criterion is graded MET or UNMET -- most of the reported numbers are redundant: Pearson's $r$, Spearman's $ρ$, Kendall's $τ_b$, the phi coefficient $φ$, and the Matthews Correlation Coefficient all reduce to a single number on non-degenerate binary data, so reporting several of them only creates an illusion of corroborating evidence. Cohen's $κ$ is the one agreement coefficient that adds information: it shares $φ$'s numerator but normalizes differently, and the gap between them measures how far the judge's positive-label rate has drifted from the human's. We then trace what changes when a judge may abstain with a CANNOT_ASSESS verdict: the three common ways of handling abstentions are not interchangeable preprocessing choices but answer different questions, and they break the binary equivalences. The same equivalences reappear, up to a negligible finite-sample correction, for multi-judge ensembles scored with Fleiss' $κ$ or Krippendorff's $α$. We close with a reporting checklist that names the judgment scale, the abstention and tie handling mode, coverage, the confusion matrix, and the aggregation level alongside any scalar agreement coefficient.

Problem

• LLM-as-Judge
• agreement metrics
• binary evaluation
• abstention handling
• human annotation

Innovation

• LLM-as-Judge
• agreement metrics
• Cohen's kappa
• binary evaluation
• abstention handling