VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision

2026-09-30

AI Summary

This study addresses the limitation of existing vision-language model benchmarks that focus solely on holistic preferences, often causing models to select images based on erroneous rationales. To this end, we construct the first contrastive image evaluation benchmark grounded in named visual criteria, introducing a novel criterion-conditioned visual discrimination paradigm. Methodologically, we design a six-axis, 51-leaf visual criterion taxonomy alongside a hidden metadata evaluation protocol. We further fine-tune Gemma-4-E4B using Direct Preference Optimization on symmetric evidence pairs, enabling multi-model evaluation through large-scale human annotation. Experimental results demonstrate that the best-performing model achieves 63.1% accuracy while significantly mitigating last-option bias. These findings reveal reliability disparities across different visual criteria, ensuring that judgment rationales remain explicit and verifiable.

Abstract

Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them. Yet existing benchmarks score only scalar quality or overall preference, so a judge can be rewarded for picking the preferred image for the wrong visual reason. We introduce VisionQ, the first benchmark built from peer-reviewed CV comparison figures that grounds every judgment in a named visual criterion: each question states the criterion, and a judge is credited only when it selects the output the authors identify as best on that criterion. We call this task criterion-conditioned visual discrimination. VisionQ comprises (1) a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures and 3,911 hand-annotated data points linking method crops to author-stated visual claims; (2) a six-axis, 51-leaf taxonomy of the visual criteria behind qualitative judgment; (3) a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity and reports accuracy per criterion; and (4) VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs, which reduces last-option predictions by 7.0pp and improves accuracy by 2.5pp on a held-out test set. Evaluating 20 open- and closed-source VLM judges, we find that the strongest reach only 63.1% accuracy (chance 32.2%) and that reliability varies sharply across criteria. Code: https://github.com/ReML-AI/visionq. Data: https://huggingface.co/datasets/visionq-anon-2026/VisionQ-1k.

Problem

• Vision-Language Models
• Qualitative Analysis
• Computer Vision Benchmark
• Visual Discrimination
• VLM-as-a-Judge

Innovation

• criterion-conditioned visual discrimination
• VLM-as-a-Judge
• qualitative comparison benchmark
• visual criteria taxonomy
• DPO fine-tuning