SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images

2026-06-10

AI Summary

This work proposes SalArt-VQA, the first fine-grained evaluation framework for assessing visual language models’ (VLMs) understanding of artifacts in AI-generated images. Comprising 950 images and 3,681 human-authored multiple-choice questions, the benchmark systematically evaluates VLMs across four dimensions: artifact existence, semantic localization, spatial localization, and evidential reasoning, while incorporating both real and generated reference images to analyze model calibration and abstention behavior. Experiments on 20 VLMs reveal that although the strongest model achieves a 99.37% recall in artifact detection, it answers all four question types correctly for only 53.26% of images. The study further uncovers a trade-off between sensitivity and calibration: highly sensitive models tend to false positives, whereas conservative ones exhibit missed detections, exposing failure modes masked by high aggregate accuracy.

Abstract

Vision-language models (VLMs) are increasingly used to detect whether AI-generated images contain visible artifacts, yet their ability to analyze such artifacts remains poorly understood. A correct image-level decision can still hide important failures: a model may correctly flag an artifact while relying on the wrong visual cue, selecting the wrong region, or describing a defect that the image does not support. To evaluate these behaviors directly, we introduce SalArt-VQA, a diagnostic benchmark for fine-grained SALient ARTifact understanding in AI-generated images. SalArt-VQA contains 950 images and 3,681 human-authored multiple-choice questions spanning artifact images, matched real reference images, and paired generated reference images. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, while the reference splits test calibration and abstention when the annotated defect is absent. Across 20 VLMs, SalArt-VQA reveals failures that image-level detection accuracy hides: the strongest model reaches 99.37% detection recall on artifact images but answers all four artifact-side questions correctly on only 53.26% of images. Comparing artifact images with artifact-free references reveals a sensitivity-calibration tradeoff: sensitive models often make unsupported artifact claims, while conservative models avoid false alarms largely by missing real artifacts. These results show that high artifact detection accuracy alone does not imply grounded artifact understanding. SalArt-VQA exposes these hidden failure modes and provides a fine-grained evaluation of whether VLM artifact claims are supported by local visual evidence.

Problem

• VLMs
• AI-generated images
• artifacts
• visual grounding
• diagnostic evaluation

Innovation

• SalArt-VQA
• vision-language models
• artifact understanding
• fine-grained evaluation
• evidence grounding