Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models

2026-09-14

AI Summary

研究通过引入H$^2$V-M方法，对视觉-语言及视觉-语言-动作模型在图像扰动下的鲁棒性进行全面验证，揭示了模型鲁棒性主要受扰动类型影响。

Abstract

Vision-language models (VLMs) and vision-language-action models (VLAs) are increasingly deployed in real-world applications. There, a small perturbation to the recorded camera image may change a decision significantly. However, existing benchmarks for these models only sample perturbations, which does not guarantee the absence of a failure in the untested region. We present the first robustness validation of six VLMs (drawn from the Gemma, InternVL, LLaVA, and Qwen families) and five VLAs (drawn from the GR00T, OpenVLA, and $π$ families) over entire continuous regions of photometric and geometric image perturbation: brightness shifts, camera rotations, and their composition. To this end, we build on the validation framework H$^2$V and introduce H$^2$V-M, a margin-aware convergence rule that makes validation affordable at the 32B parameter scale. We demonstrate that H$^2$V-M outperforms H$^2$V by an order of magnitude in model queries and that it finds counterexamples faster than random sampling while providing soundness guarantees. Our VLM and VLA robustness validation shows that robustness is mostly dependent on the perturbation type, rather than the model, and that VLMs are more robust to large camera rotations than VLAs. For VLAs, even perturbations as small as $\pm1^\circ$ can change the commanded action in many cases. We also show that robustness depends more on model family than on model size.

Problem

• Vision-Language Models
• Robustness
• Perturbation
• Validation
• Benchmark

Innovation

• robustness validation
• H$^2$V-M
• continuous regions of perturbation
• margin-aware convergence rule