Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning

2026-07-30

AI Summary

This work addresses the performance bottleneck in strict zero-shot image captioning caused by the absence of visual feedback during inference. The authors propose a multi-agent framework that enhances vision-language alignment without retraining existing captioners, leveraging multi-stage alignment scoring and unsupervised consensus distillation. A novel multi-checkpoint visual feedback mechanism is introduced during decoding, accompanied by a lightweight learnable reranking module that integrates TriFuse and MemAttend architectures. The approach further incorporates Borda count–based consensus distillation to refine caption selection. Experimental results demonstrate significant improvements: the method achieves a CIDEr score of 117.6 on the COCO Karpathy test set, surpassing the baseline by 9.6 points, and yields gains of 8.1 and 5.7 on Flickr30k and NoCaps, respectively.

Abstract

Zero-shot image captioning (ZIC) describes images without paired image-caption supervision during captioner training, relying on text-only corpora and frozen pretrained image-text scorers. Existing retrieval-augmented methods score image-text alignment once, at retrieval, then commit the captioner's autoregressive beam under language-model probability alone, leaving the decoder without further visual grounding feedback. Progress has stalled, with no method improving on the strict-regime best since 2024. We propose Adjudicated Captioning, an inference-time multi-agent framework that restores grounding feedback at multiple checkpoints over an unchanged IFCap captioner. First, we install a stronger frozen Retrieval Encoder at the input. Second, between retrieval and decoding we insert a frozen Cross-Attention Verifier that re-ranks the top-9 retrievals to top-5. Third, at the output beam we attach a learned Reranker pairing TriFuse, a multilayer perceptron, with MemAttend, a memory-attended transformer, the pipeline's only learned components; both are trained self-supervised by Borda-consensus distillation across the three frozen scorers, using no paired image-caption labels and no reference captions. Under the inductive headline protocol, with rerankers fit on the disjoint COCO Karpathy validation beam and applied frozen to test, the framework reaches CIDEr 117.6 and SPICE 21.9 on COCO Karpathy, up from 108.0 and 20.3 for IFCap, a +9.6 CIDEr gain, and +7.7 above NES, the strongest synthetic-image-augmented method at 109.9, without retraining the captioner. A training-free fixed-fusion baseline reaches 115.8 CIDEr, so +7.8 of the +9.6 gain comes from the non-learned architectural intervention and the remaining +1.8 from the learned rerankers. The same recipe transfers off-COCO without captioner retraining: +8.1 CIDEr on Flickr30k Karpathy and +5.7 on NoCaps overall.

Problem

• zero-shot image captioning
• image-text alignment
• visual grounding
• strict-regime captioning
• retrieval-augmented generation

Innovation

• Zero-shot image captioning
• Multi-agent alignment scoring
• Consensus-distilled reranking
• Frozen scorer arbitration
• Training-free transfer