TPCap: Unlocking Zero-Shot Image Captioning with Trigger-Augmented and Multi-Modal Purification Modules

2025-02-16

AI Summary

Existing retrieval-augmented generation (RAG)-based image captioning methods rely on external retrieval corpora, compromising large language models’ (LLMs’) zero-shot capability and incurring substantial computational overhead. This paper proposes TPCap, the first framework for zero-shot image captioning without any external retrieval database. Its core comprises two synergistic modules: Trigger Augmentation (TA), which employs a learnable trigger projector to activate the LLM’s contextual reasoning ability, and Multimodal Purification (MP), which aligns visual–textual representations while suppressing cross-modal noise. With only 0.82M trainable parameters, TPCap is lightweight and factually consistent. It achieves state-of-the-art performance across four benchmarks—COCO, NoCaps, Flickr30k, and WHOOPS—outperforming all prior methods. Moreover, it enables efficient training and inference on a single RTX 4090 GPU.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent advancements in large language models (LLMs) have significantly enhanced the fluency and logical coherence of image captioning. Retrieval-Augmented Generation (RAG) is widely adopted to incorporate external knowledge into LLMs; however, existing RAG-based methods rely on separate retrieval banks, introducing computational overhead and limiting the utilization of LLMs' inherent zero-shot capabilities. To address these limitations, we propose TPCap, a novel trigger-augmented and multi-modal purification framework for zero-shot image captioning without external retrieval libraries. TPCap consists of two key components: trigger-augmented (TA) generation and multi-modal purification (MP). The TA module employs a trigger projector with frozen and learnable projections to activate LLMs' contextual reasoning, enhance visual-textual alignment, and mitigate data bias. The MP module further refines the generated entity-related information by filtering noise and enhancing feature quality, ensuring more precise and factually consistent captions. We evaluate TPCap on COCO, NoCaps, Flickr30k, and WHOOPS datasets. With only 0.82M trainable parameters and training on a single NVIDIA RTX 4090 GPU, TPCap achieves competitive performance comparable to state-of-the-art models.

Problem

• Enhance zero-shot image captioning
• Reduce computational overhead
• Improve visual-textual alignment

Innovation

• Trigger-Augmented Generation
• Multi-Modal Purification
• Zero-Shot Capabilities