CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

2026-08-03

AI Summary

This work addresses the limitation of conventional image captioning evaluation metrics, which conflate visual content coverage and factual accuracy into a single scalar score. To resolve this, the authors introduce CAPEval, a novel benchmark that decouples caption quality into two distinct dimensions: Coverage—measuring the completeness of grounded visual content—and Precision—assessing the correctness of stated facts—using human-annotated, atomized fact-checking checklists. Through controlled end-to-end downstream task experiments, the study demonstrates that Coverage is more strongly correlated with multimodal understanding performance, whereas Precision plays a dominant role in text-to-image generation quality. These findings provide both theoretical grounding and practical guidance for selecting or designing captioning systems tailored to specific downstream applications.

Abstract

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

Problem

• caption evaluation
• coverage
• precision
• multimodal understanding
• text-to-image generation

Innovation

• decoupled evaluation
• caption quality
• coverage
• precision
• multimodal understanding