🤖 AI Summary
Current AI evaluation benchmarks often report numerous scores without verifying their informational independence, leading to redundancy and potential misinterpretation. This work proposes Effective Dimensionality (ED) as a diagnostic metric to quantify the informational breadth of benchmarks, enabling the first rapid estimation of an upper bound on the number of independent evaluation dimensions under population-level conditions. Leveraging 22 benchmarks across 8 domains and over 8,400 model evaluations, we develop a four-step diagnostic pipeline and reference atlas. ED is computed via the participation ratio of centered score spectra and validated through null models, reliability and saturation analyses, and controlled experiments. Our findings reveal substantial redundancy in widely used benchmarks: the Open LLM Leaderboard effectively captures only about 1.7 independent dimensions, while BBH and MMLU-Pro are highly interchangeable (ρ = 0.96), with benchmark informational breadth varying by more than twentyfold.
📝 Abstract
AI evaluation suites often report many scores without checking whether those scores carry independent information. We introduce Effective Dimensionality (ED), the participation ratio of a centered benchmark-score spectrum, as a fast, population-conditional upper-bound diagnostic of measurement breadth. Applied at per-instance granularity to 22 benchmarks across 8 domains and more than 8,400 model evaluations, ED reveals substantial redundancy: the six-score Open LLM Leaderboard behaves like roughly two effective measurement axes (ED = 1.7), BBH and MMLU-Pro are near-interchangeable (rho = 0.96, stable across seven subpopulations), and measurement breadth varies more than 20x across current benchmarks. We show that relative ED rankings are stable under matched-dimension controls and that ED can flag redundant suite components, monitor performance-conditional compression, and guide benchmark maintenance. Because binary spectra overestimate absolute latent dimensionality, we interpret ED as a screening statistic rather than a literal factor count and complement it with null, reliability, and saturation analyses. We provide a 22-benchmark reference atlas and a four-step diagnostic workflow that benchmark maintainers can run with a score matrix and a few lines of code.