🤖 AI Summary
This study addresses the absence of preregisterable minimum detectable effect (MDE) standards in quantized model evaluation, which hinders disentangling quantization noise from other sources of variation. Adapting paired binomial sample size methodology to 4-bit quantization benchmarks, the work proposes a conservative upper-bound formula for MDE to enable effect-size budgeting in experimental design and validates its efficacy through pilot audits. Innovatively integrating a preregistrable MDE framework with the Miettinen test, FP16–NF4 disagreement rate modeling, cross-model and cross-benchmark experiments, and variance decomposition by prompt template, the analysis reveals that most NF4–FP16 performance differences fall below the MDE threshold. In contrast, prompt templates induce performance fluctuations of 2–10 percentage points—substantially exceeding quantization effects—thereby underscoring the critical need to control for prompt-induced variability.
📝 Abstract
This is a planning-method note with an unpaired pilot audit. We adapt the classical paired-binary sample-size calculation (Miettinen, 1968) to quantization benchmarks, giving a conservative minimum detectable effect (MDE) bound $δ^{*} \le (z_{1-α/2}+z_{1-β})\sqrt{ρ_d/m}$ in the paired item count $m$ and the FP16-NF4 disagreement rate $ρ_d$. The bound turns "how reliable is my quantization claim?" into a one-line budget a benchmark designer can commit to before running. We illustrate the bound on four models and four benchmarks ($k=5$ splits of $n=100$), and add a parallel MMLU prompt-template study to put the bound's quantization-noise scale alongside the prompt-noise scale. Assuming $ρ_d=0.10$ (an unmeasured planning value), all observed NF4-FP16 deltas fall below the implied MDE, and most cross-split SDs lie within $\pm 1.5$ pp of the binomial reference $\sqrt{p(1-p)/n}$, so much of the variance reported as "benchmark unreliability" on $n=100$ subsamples is binomial sampling noise. The single borderline cell (OPT-WinoGrande, $|Δ|=3.2$ pp) is below the implied MDE at $ρ_d=0.10$ but above it at $ρ_d=0.05$, illustrating the planning trade-off the bound makes explicit. On MMLU, prompt-template ranges of 2-10 pp meet or exceed the largest observed quantization delta (3.2 pp), so a quantization audit that does not first fix the prompt template absorbs template variance into its noise floor. We complement the bound with a five-line pre-registration template.