RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation

2026-08-03

AI Summary

This work addresses a critical limitation in current LLM-as-Judge evaluation frameworks, which assume independence among scoring dimensions and thereby overlook inherent behavioral couplings that distort aggregated scores. To mitigate this, the authors propose RADAR, a lightweight pre-evaluation diagnostic framework that generates targeted synthetic probes to model inter-dimensional dependencies and redundancies prior to large-scale assessment. RADAR outputs a directed coupling matrix that reveals covariation patterns among criteria, introducing— for the first time—a coupling-aware analysis mechanism into LLM evaluation pipelines. Empirical validation on HelpSteer2, SumPubMed, and SummEval benchmarks demonstrates that RADAR accurately reconstructs human annotators’ inter-dimensional correlation structures (Pearson r > 0.84) using only a small number of probes, thereby delivering actionable audit signals regarding redundancy, hierarchical relationships, and aggregation sensitivity.

Abstract

Rubric-based LLM-as-judge pipelines often assume that evaluation criteria provide independent signals. In practice, however, criteria can be behaviorally coupled: improving one criterion may systematically change scores on another, distorting aggregate scores used in model-release or product-update decisions. We introduce RADAR, a lightweight preflight diagnostic framework for estimating such coupling before large-scale evaluation. Given a rubric, RADAR generates targeted synthetic probes, scores each probe on all criteria, and produces a directional coupling matrix that shows which criteria co-score and how. We validate RADAR on three industry-relevant evaluation settings: NVIDIA HelpSteer2, SumPubMed, and the Yale-Salesforce SummEval benchmark. Using only a small number of probes per criterion, RADAR recovers human inter-criterion correlation structure (Pearson r > 0.84) and provides practitioners with concrete audit signals about redundancy, hierarchy, and aggregation sensitivity before committing to large-scale judging.

Problem

• LLM-as-judge
• evaluation rubric
• criterion coupling
• score redundancy
• aggregate evaluation

Innovation

• LLM-as-judge
• rubric analysis
• criterion coupling
• synthetic probing
• evaluation diagnostics