It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling

2026-08-02

AI Summary

This work addresses a critical limitation in existing self-consistency selection methods like PGS, which struggle to distinguish image-dependent answers from language-prior guesses during test-time scaling of vision-language models. To rigorously evaluate PGS under controlled conditions, the authors propose MatchedCtrl—a format-matched control approach that isolates the effect of decoding format while holding computational budget constant. Using simple, label-free, and training-free perturbations (e.g., cropping, background masking, and mild photometric or geometric distortions), they demonstrate for the first time that the observed performance gains stem primarily from the decoding format rather than the perturbations themselves. Across benchmarks including TextVQA, MATH-Vision, MMMU, and ViLP, MatchedCtrl matches or exceeds PGS within noise margins, revealing no significant advantage for PGS and thereby challenging prior claims about its efficacy in diagnosing visual dependence.

Abstract

Test-time scaling lifts large language model reasoning by sampling many candidate solutions and selecting among them, yet the same recipe transfers poorly to vision-language models (VLMs): recent work shows that simple majority voting beats selection methods built on the model's own self-verification, apparently because at the selection layer an image-grounded answer and a confident guess from the language prior look the same. A natural fix is to make the selection signal one that cannot be computed without the image. We study Perturbation Grounded Selection (Pgs), a label-free, training-free rule that scores each candidate by whether the model re-derives it under label-preserving perturbations of the input (cropping, background masking, mild photometric or geometric jitter); Pgs recovers majority voting when the perturbation set is empty. The decisive question is not whether Pgs beats chain-of-thought only majority voting, but whether the perturbation term adds anything once decoding format and budget are controlled. We therefore introduce a format-matched control (MatchedCtrl): the same short, no-CoT draws spent on the original image. Across TextVQA, MATH-Vision, MMMU, and ViLP, with a Qwen headline (three-seed means) and LLaVA-OneVision coverage in matched-budget selector tables, Pgs appears to beat plain majority voting by up to +31.8 points on TextVQA (Qwen), but MatchedCtrl tracks or exceeds Pgs within noise on every benchmark, including the vision-required ViLP; no Qwen category shows a significant gain over this control. The stability gap is real and image-dependent (up to +0.48), yet does not predict per-instance wins. The result is negative and diagnostic: perturbation consistency is at best a partial diagnostic of visual dependence and, on its own, not a usable selection signal once format is controlled; gains reported against CoT-only majority voting overstate such methods.

Problem

• vision-language models
• test-time scaling
• selection consistency
• perturbation
• visual dependence

Innovation

• test-time scaling
• vision-language models
• perturbation consistency
• decoding format
• selection mechanism