Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

2026-06-08

AI Summary

This study investigates whether pairwise comparisons genuinely reflect model accuracy or are confounded by stylistic preferences and annotator biases. The authors reformulate five standard benchmarks as open-ended generation tasks and integrate pairwise comparisons, Elo rating aggregation, and Spearman correlation analysis with causal inference techniques. Their systematic evaluation—conducted in settings where ground-truth labels are available—demonstrates for the first time that Elo rankings exhibit strong alignment with accuracy-based rankings (Spearman correlation > 0.9), significantly outperforming direct scoring, especially under weak annotator conditions. Furthermore, they identify “answer repetition at the end” as a key causal factor driving human preference, while finding that stylistic variation and annotator bias exert only minimal influence on overall model rankings.

Abstract

Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. By converting five well-known benchmarks into free-form generative evaluations, we find that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings and substantially outperform direct evaluation when the judge is weak. Furthermore, style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct (or incorrect). On such pairs, we find that repetition after the final answer (echo) is a causal driver of judge preference.

Problem

• pairwise comparisons
• generative models
• accuracy rankings
• judge bias
• Elo ranking

Innovation

• pairwise comparisons
• Elo ranking
• generative model evaluation
• accuracy correlation
• judge bias