SABRE: Scalable and Automated Benchmarking of VLMs under Stress

2026-08-07

AI Summary

This work addresses the limitations of existing vision-language model (VLM) benchmarks in effectively uncovering model weaknesses under stress conditions and the high cost of manually curating high-quality test samples. To this end, we propose SABRE, a novel framework that enables end-to-end automated generation of stress-test datasets directly from task design specifications. SABRE establishes a human-in-the-loop data production pipeline through structured specification parsing, image generation and editing, automated filtering, and human validation. The framework supports multidimensional stress testing—such as context, texture, and attribute manipulation—and is both extensible and reusable. Using SABRE, we construct the SABRE-Prior benchmark (600 images, 1,000 questions), on which six state-of-the-art VLMs achieve an average accuracy of only 22.6%, revealing their overreliance on world priors rather than visual evidence. The framework also demonstrates strong transferability across tasks like counting and spatial reasoning.

Abstract

Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.

Problem

• vision-language models
• stress testing
• benchmarking
• scalability
• automated evaluation

Innovation

• SABRE
• vision-language models
• stress testing
• automated benchmarking
• test generation