Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction

2025-12-03

AI Summary

This work addresses the bottleneck in unsupervised image captioning—its reliance on manually aligned image–text pairs—by proposing TOMCap, a training-free method that leverages only raw text corpora. Methodologically, TOMCap jointly exploits CLIP’s cross-modal representations and retrieval-augmented prompting: it retrieves semantically relevant textual exemplars and aligns them with image latent vectors in CLIP’s shared embedding space; a modality-gap correction mechanism further guides a pretrained language model decoder to generate high-quality captions. This approach effectively bridges the vision–language modality gap, enabling zero-shot image captioning. Extensive experiments on Flickr30K and COCO demonstrate that TOMCap significantly outperforms existing training-free and text-only baselines. The results validate both the efficacy and generalizability of retrieval augmentation coupled with latent-space alignment for unsupervised multimodal generation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Image captioning has drawn considerable attention from the natural language processing and computer vision fields. Aiming to reduce the reliance on curated data, several studies have explored image captioning without any humanly-annotated image-text pairs for training, although existing methods are still outperformed by fully supervised approaches. This paper proposes TOMCap, i.e., an improved text-only training method that performs captioning without the need for aligned image-caption pairs. The method is based on prompting a pre-trained language model decoder with information derived from a CLIP representation, after undergoing a process to reduce the modality gap. We specifically tested the combined use of retrieved examples of captions, and latent vector representations, to guide the generation process. Through extensive experiments, we show that TOMCap outperforms other training-free and text-only methods. We also analyze the impact of different choices regarding the configuration of the retrieval-augmentation and modality gap reduction components.

Problem

• Reduces reliance on human-annotated image-caption pairs
• Corrects modality gap between text and image representations
• Enhances captioning using retrieval-augmented text-only training

Innovation

• Text-only training with retrieval-augmented caption examples
• Modality gap reduction for CLIP-guided language model prompting
• Training-free image captioning without aligned image-text pairs