🤖 AI Summary
This work investigates the performance degradation of verifier-driven self-improving vision-language models in cross-task settings, where task-specific verifiers often hinder generalization. The study systematically demonstrates that even highly accurate verifiers can significantly impair student model performance—by 3.4 to 10.9 percentage points—when their task alignment is poor, and introduces a “directional mismatch” mechanism to explain this phenomenon. Building on frozen verifier scores, preference pair construction, and DPO training, the authors integrate variance analysis with a progress-gated replay strategy to derive practical guidelines for verifier selection at deployment time. Empirical validation on benchmarks such as MMMU confirms the findings, with consistent results reproduced across two Qwen-VL model variants.
📝 Abstract
Verifier-driven self-DPO is a common recipe for self-improving production visual-language models. In this setup, a frozen verifier scores candidate generations, the top- and bottom-scoring candidates form a preference example, and DPO updates the learner. The deployment-time assumption is monotone: a stronger verifier should yield a stronger student. We show that this assumption can fail because verifier quality is highly task-specific. On a four-rung open-source verifier ladder across MathVista, MMMU, and BLINK, the same verifiers that are above-threshold and improve a Qwen-3-VL-2B student on MathVista become sub-threshold on MMMU, where their task-rubric accuracy drops to 8% to 23%.
In this regime, every verifier we tested silently regresses the student, producing drops of 3.4 to 10.9 percentage points below the frozen baseline while the DPO training loss continues to decrease. The regression replicates on a second student, Qwen-2.5-VL-3B. Moreover, within the failure regime, damage is confidence-inverted: the more accurate-but-still-wrong verifier causes larger regression than a near-random verifier, suggesting that progress-gated replay amplifies confidently wrong preference pairs. We give a compact mechanistic explanation via a variance theorem for progress-gated replay and its direction-mismatch failure mode.
The deployment message is operational rather than purely diagnostic: before running any verifier-driven loop, teams should measure target-task rubric accuracy, rank verifiers by target-task rubric quality rather than parameter count, and treat diminishing returns in above-threshold regimes as a verifier-side compute budget cap.