The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale

2026-08-04

AI Summary

This work addresses the widespread misattribution of accuracy gains from language model self-correction to enhanced reasoning, demonstrating that such improvements often stem primarily from answer format repair rather than genuine content refinement. The authors propose disentangling self-correction effects into content and format margins and introduce the notion of a “calibration lower bound.” Through causal intervention experiments, multi-model comparisons, constrained decoding, parseability analysis, and statistical testing across 12 high-unparseability benchmarks, they find that self-correction in small-to-medium models is predominantly driven by format effects—significantly outweighing content contributions. Causal interventions eliminate, on average, 71% of the estimated performance gain, while large models exhibit near-zero content margins, revealing substantial confounding in current self-correction evaluations.

Abstract

Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally. Across Qwen3.5 (0.8B-9B), Gemma-4-12B, and two frontier models via API (Tencent Hy3, Nvidia Nemotron-3-Ultra-550B) in 29 primary cells plus a frontier arm, we decompose the always-revise accuracy shift into a content margin (both answers parseable) and format-recovery/loss margins (parseability changes). On 12 cells with meaningful unparseable-answer rates, format effects exceed content effects (Wilcoxon p=1.7e-3). To test this causally, we force already-generated reasoning through grammar-constrained decoding so every answer is parseable by construction: across 14 cells this closes a median 71% of the gap between the naive total effect and the content-margin estimate, with two cells converging exactly and a residual on the two largest-effect cells reported rather than dismissed. A clustered model confirms floor-scale (0.8B/2B) models have far higher odds of content-level change and harm than capable-scale models (p<1e-7). Replicating a cited confidence-gating protocol verbatim on Qwen3.5 does not reproduce its reported gain and shows the same near-zero content margin. A frontier check on much larger models shows format-dominance intensifying with scale: content margin is exactly zero in all 5 cells despite total effects up to +0.275, though this arm is lower-powered. The calibration-floor criterion on the content margin reveals a squeeze: floor-scale cells have headroom but insufficient signal, capable-scale cells have signal but little headroom; only one cell is marginally viable, with negligible sealed-holdout gain. Content is a minority share of what the field has measured as self-correction. We release the instrument, code, and derived results.

Problem

• self-correction
• format repair
• answer extraction
• calibration floor
• parseability

Innovation

• self-correction
• format repair
• grammar-constrained decoding
• calibration floor
• content margin