🤖 AI Summary
This study investigates the capacity of large language models (LLMs) to serve as automated evaluators for assessing response accuracy in retrieval-augmented generation (RAG) and agent-based systems, specifically their ability to replicate human judgments. We propose a two-stage evaluation framework that systematically benchmarks 54 LLMs against human annotations using Pearson correlation, Cohen’s Kappa, and z-score metrics. Critically, we argue that correlation alone is insufficient for evaluator validation and introduce the “Judge Turing Test”—a novel paradigm prioritizing inter-judge consistency—and establish a standardized, hierarchical benchmark for discriminating LLM judging capabilities. Results show that 27 models achieve top-tier performance: 23 exhibit human-like judgment consistency, while 4 surpass human inter-annotator agreement. Crucially, model performance correlates more strongly with training methodology than with parameter count, challenging prevailing scale-centric assumptions in evaluator design.
📝 Abstract
This research introduces the Judge's Verdict Benchmark, a novel two-step methodology to evaluate Large Language Models (LLMs) as judges for response accuracy evaluation tasks. We assess how well 54 LLMs can replicate human judgment when scoring responses from RAG (Retrieval-Augmented Generation) or Agentic pipelines against ground truth answers. Our methodology progresses from traditional correlation analysis to comprehensive Cohen's Kappa analysis that measures actual agreement patterns. The two-step approach includes: (1) a correlation test that filters judges with strong alignment, followed by (2) a human-likeness test using z-scores to identify two distinct judgment patterns: human-like judgment (|z| < 1) that mimics natural human variation, and super-consistent judgment (z > 1) that exceeds typical human-to-human agreement levels. This methodology reveals that 27 out of 54 tested LLMs achieve Tier 1 performance: 23 models exhibit human-like patterns that preserve the nuances of human judgment, while 4 models demonstrate super-consistent behavior, a pattern that could indicate either enhanced reliability or oversimplification of complex judgments. Testing 43 open-source models (1B-405B parameters) and 11 closed models (GPT, Gemini, Claude variants), we demonstrate that judge excellence is not solely dependent on model size but on specific training strategies. Our key contributions include: (1) establishing that correlation alone is insufficient for judge evaluation, (2) introducing a "Turing Test for judges" based on agreement patterns, and (3) providing a standardized benchmark for classifying LLM judges into distinct performance tiers for different evaluation needs.