A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions

2026-07-25

AI Summary

This work addresses the limitations of existing image captioning evaluation methods, which predominantly rely on human-annotated reference descriptions and struggle to assess semantic faithfulness. To overcome this, the authors propose a reference-free evaluation framework that judges caption quality through semantic-equivalent reconstruction: captions are used to reconstruct images, and the reconstructed images are evaluated based on their performance consistency with the original images in downstream vision-language tasks. Departing from pixel-level reconstruction, this approach introduces a novel evaluation principle centered on semantic equivalence and incorporates a task-conditioned scoring mechanism. Leveraging a newly constructed Captioning Turing Test Dataset (CTTD), the study establishes the first reference-free evaluation system that effectively captures the semantic fidelity of captions while substantially reducing annotation costs.

Abstract

Image captioning is a primary task in vision--language research, yet assessing how faithfully a caption preserves image semantics without relying on reference captions remains unsettled. Prevailing evaluations rely on human-annotated references, whose content reflects annotator intent and captioning proficiency. In this paper, we study a reconstruction-based principle for caption evaluation: a caption is as good as its capacity to enable reconstruction of the original image. However, because captioning inherently compresses visual information, it is impossible to recover all details, and pixel-wise comparison between reconstructed and source images is neither feasible nor meaningful. Through our in-depth analysis of the nature of captions, whose fundamental purpose is to transmit the semantic content of an image, we propose a revised principle: a caption is as good as its capacity to enable a reconstruction that is semantically equivalent to the original. To assess semantic equivalence, we test whether the reconstruction matches the original image across a suite of downstream vision--language tasks, yielding a reference-free, task-conditioned caption score. We characterize component-dependent limitations and introduce the lower-cost Captioning Turing Test Dataset (CTTD) surrogate.

Problem

• caption evaluation
• reference-free
• semantic equivalence
• image captioning
• vision-language tasks

Innovation

• reconstruction-based evaluation
• reference-free captioning
• semantic equivalence
• vision-language tasks
• Captioning Turing Test Dataset