🤖 AI Summary
This study addresses the limitations of current LLM-as-a-Judge evaluation paradigms, which rely heavily on uncalibrated exact-match metrics that overstate models’ discriminative capabilities by ignoring random agreement. Through a systematic assessment of 21 judge models from nine providers—spanning 118 experiments and approximately 541,000 judgments across three major benchmarks—the work introduces Cohen’s kappa as a more robust alternative to exact match, implements cross-benchmark evaluation, quantifies position and verbosity biases, and conducts high-density test-retest reliability analyses. The findings reveal a critical disconnect between reliability and validity: kappa scores drop by 33–41 percentage points relative to exact-match accuracy, and judge rankings shift by up to 14 positions across benchmarks. The study further proposes a minimal viable validation protocol and identifies universal patterns such as the “consistency–bias paradox” across diverse models.
📝 Abstract
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.