🤖 AI Summary
Current AI benchmarks rest on unexamined theoretical assumptions, leading to self-reinforcing evaluation frameworks that obscure the structural limitations of dominant paradigms. This work proposes “Epistematics”—a novel meta-evaluation framework that derives assessment criteria directly from claims about technical capabilities, thereby auditing whether benchmarks effectively distinguish target competencies from proxy behaviors and ensuring alignment between evaluation protocols and the underlying definitions of capability. Integrating philosophical and computational perspectives, the framework comprises an auditing procedure, a taxonomy of failure modes, and design principles for benchmark construction, enabling both logical and empirical scrutiny of evaluation systems. Applied to the proposal by Dupoux et al. (2026), the analysis reveals how architectural innovations were undermined by inadequate evaluation criteria, inadvertently reinforcing existing constraints and demonstrating the framework’s efficacy in exposing misalignments between theory and assessment.
📝 Abstract
Every AI benchmark operationalizes theoretical assumptions about the capability it claims to assess. When assumptions function as unexamined commitments, benchmarks stabilize the dominant paradigm by narrowing what counts as progress. Over time, narrow evaluation reorganizes capability concepts: architectures and definitions are selected for benchmark legibility until evaluation ceases to track an independent object and instead produces a version of the target defined by its own operational assumptions. The result is a trap: evaluation frameworks treat self-reinforcing assessments as valid, both creating and obscuring structural limits on what the current paradigm can accomplish. We introduce Epistematics, a methodology for deriving evaluation criteria directly from technical capability claims and auditing whether proposed benchmarks can discriminate the claimed capability from proxy behaviors. The contribution is meta-evaluative: an audit procedure, a failure mode taxonomy, and benchmark-design criteria for evaluating capability-evaluation coherence. We demonstrate the procedure through a worked audit of Dupoux et al. (2026), a proposal that revises the dominant paradigm's theoretical assumptions at the architectural level while reproducing them in its evaluation criteria, thereby entrenching the constraint it seeks to overcome in a form the evaluation cannot detect.