DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

2026-10-02

AI Summary

This study addresses the limitations of existing text-to-image evaluation metrics, including poor generalization, a lack of fine-grained assessment, and misjudgments caused by reliance on fixed reference answers. To overcome these issues, this work proposes DEPICT, a training-free metric that discards static references in favor of a dynamic scoring mechanism based on expected consistency. Specifically, DEPICT leverages vision-language models to evaluate consistency through independent visual and textual question answering, employing multi-granularity weighted fusion to recover contextual information and effectively resolve the longstanding challenge of evaluating negation prompts. Extensive experiments across five benchmarks and eleven backbone networks demonstrate that DEPICT comprehensively outperforms existing training-free metrics and surpasses fine-tuned evaluators on two human-correlation benchmarks.

Abstract

Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a vision-language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation benchmarks.

Problem

• Text-to-Image Alignment
• Evaluation Metrics
• Text-to-Image Generation
• Hallucination Detection

Innovation

• Text-to-Image Alignment
• Training-free Metric
• Answer Agreement
• Negation Accuracy
• Vision-Language Model