Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

2026-08-07

AI Summary

This work addresses the limited efficacy of conventional aggregation methods in large language model (LLM) ensembles, which often stems from highly correlated errors and yields only marginal global gains from external verification signals. The authors propose a utility-aware, decision boundary–based hierarchical analysis framework and demonstrate that the benefits of verification are concentrated on critical “one-vote-away” queries—those where a single vote determines the ensemble outcome. By applying a single-vote replacement strategy coupled with marginal error analysis, and integrating majority voting with subsampled verification across benchmarks (HumanEval+/MBPP+), the method invokes verification signals for only 16.2% of these pivotal queries. This targeted approach improves accuracy on these samples by 10.4–23.3 percentage points and elevates overall ensemble accuracy from 82.44% to 85.62%.

Abstract

LLM judge panels are a standard evaluation tool, but prior work reports highly correlated panel errors: nine judges provide roughly the effective information of two independent ones, and aggregation closes only a small fraction of the gap. A natural remedy--a signal from a different evidence source, e.g., executing a test suite--produced no distinguishable change in the panel's effective-vote count at scale (-0.04, 95\% CI [-0.10, +0.02]). Aggregate dependence and conditional decision utility are different questions. Elementary majority arithmetic fixes the affected set for single-ballot substitution: only decisions with a one-vote margin can change. The empirical question is whether panel error rates rise and useful substitutions concentrate there. They do: the entire accuracy gain concentrates on these pivotal queries, where it is large (+10.4 to +23.3 percentage points across three headline configurations), and is exactly zero elsewhere. We confirm the pattern across three code benchmarks and four panel sizes (a 9-judge extension and 56 dependent subsampling checks, gain +6.5 to +16.1 percentage points). On HumanEval+/MBPP+, a majority-side replacement rule raises overall accuracy from 82.44\% to 85.62\% while invoking the signal on 16.2\% of queries; signal-only remains stronger at 87.60\%. Thus population-level dependence diagnostics and margin-stratified utility are complementary, and the affected-set characterization yields a call-reduction rule for any specified single-ballot substitution policy.

Problem

• LLM judge panels
• aggregate independence
• pivotal vote
• conditional decision utility
• verification signal

Innovation

• pivotal vote
• aggregate dependence
• conditional utility
• majority-margin stratification
• verification signal