🤖 AI Summary
This study challenges the prevailing assumption in AI evaluation that consistency implies correctness, systematically investigating whether self-consistency and cross-model agreement reliably indicate answer accuracy in large language models. Leveraging the GPQA Diamond and AIME datasets, the authors analyze 265,000 samples across 53 model instances using hierarchical clustering with bootstrapping, majority-correct labeling, cross-model-family validation, and margin-preserving null hypothesis testing. They find that consistency serves only as a weak positive predictor of correctness (ρ = 0.20–0.59), with its efficacy strongly dependent on model capability tier and task saturation: it is most informative for mid-tier, unsaturated models but can induce overconfidence in state-of-the-art models, yielding high agreement alongside high error rates. All data are publicly released.
📝 Abstract
LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles (Verga et al., 2024) or "mixture-of-experts" (Shazeer et al., 2017) panels of judges. These systems share a key assumption: that consistency -- agreement among judges, or among a model's own samples -- indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with itself, and different models can agree with each other, out of shared bias, a memorized heuristic, or an option-position prior rather than truth. We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study: 53 runners drew K=50 samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME -- 265,000 samples. Using majority-correctness as the deployment label and a hierarchical runner-clustered bootstrap, agreement is a positive but weak predictor (rho 0.20-0.59, all positive under item-clustered resampling) whose usefulness is regime-dependent: best for unsaturated mid-tier models and for allocating compute, and worst -- over-confident yet no more accurate -- for the most consistent frontier model (agreement >=0.8 on 77% of GPQA case-result entries, 48% of those wrong). An exploratory cross-family check on three Claude tiers shows the same frontier over-confidence, with confident errors recurring across providers above a marginal-preserving null. Self-consistency is thus a conditional proxy for correctness, not a standalone confidence score. We publicly release the de-identified per-run rows and answer distributions.