Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

2026-10-06

AI Summary

This study addresses the limitation that single aggregate accuracy metrics in LLM-as-a-judge evaluations obscure performance disparities across varying deployment conditions. To this end, it introduces the Conditional Accuracy Profiling (CAP) framework, which decomposes pairwise judging accuracy into eight distinct conditions. By integrating posterior diagnostics, task-subset surrogates, and controlled augmentation techniques, CAP enables benchmark-agnostic, multidimensional performance profiling. Experiments across six benchmarks demonstrate that CAP reveals sensitivity differences unpredictable by aggregated metrics, uncovering hidden vulnerabilities such as stable positional robustness rankings that remain susceptible to adversarial attacks. Ultimately, this work provides actionable diagnostic criteria for informed judge selection.

Abstract

LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We introduce \textbf{Conditional Accuracy Profiling} (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality. CAP is benchmark-agnostic: it can be applied directly when a benchmark provides the required annotations, approximately through task-subset proxies, or through controlled augmentation when perturbation pairs can be generated. We instantiate CAP on seven LLM judges across six pairwise judging benchmarks, including \textsc{judgerEva-Standard}, a controlled testbed we created to support all eight conditions. CAP exposes profile differences hidden by aggregate accuracy: on \textsc{judgerEva}'s judge-independent Hard-Constructed subset, the two judges most sensitive to omitted qualifications rank in the bottom three of seven by overall accuracy, so omission sensitivity is not predicted by aggregate accuracy. Across benchmarks, Position Robustness shows the strongest rank stability (mean Spearman $\barρ{=}0.87$) but is itself fragile under JudgeBench-Pro adversarial stress, showing the largest mean accuracy drop among the shared conditions, though the dominant degradation channel varies by judge. Condition-level profiles provide a more actionable basis than aggregate accuracy for selecting LLM judges.

Problem

• LLM-as-judge
• evaluation
• conditional accuracy
• diagnostic profiling
• aggregate metrics

Innovation

• Conditional Accuracy Profiling
• LLM-as-judge
• Diagnostic Framework
• Benchmark-agnostic
• Pairwise Evaluation