🤖 AI Summary
This work addresses the limitations of current image and video generation models, which suffer from poor text–image alignment due to low-quality captions generated by vision-language models—characterized by hallucinations, weak compositional reasoning, and insufficient fine-grained understanding. To overcome these issues, the authors propose a dual-path optimization strategy: first, constructing a high-quality, “clean” annotated dataset through hierarchical sampling that avoids reliance on web-scraped, copyrighted content; second, enhancing caption quality via context-aware alignment and supervised fine-tuning (SFT), augmented with a fine-tuned character detection model to produce structured captions that improve downstream usability. The study also introduces the first systematic categorization of caption evaluation metrics into generic and instance-anchored types. Experiments on open-source models demonstrate significant improvements in text–image alignment, particularly in caption accuracy and structural coherence.
📝 Abstract
Caption quality has emerged as a critical bottleneck in training high-quality text-to-image (T2I) and text-to-video (T2V) generative models. While visual language models (VLMs) are commonly deployed to generate captions from visual data, they suffer from hallucinations, poor compositional reasoning, and limited fine-grained understanding, resulting in misaligned image-caption pairs that degrade downstream model performance. This technical report introduces VIVECaption, a systematic two-sided approach to caption quality improvement. We first establish a comprehensive taxonomy of caption evaluation metrics, distinguishing between"universal"and"instance-grounded"metrics, with the ultimate goal of showcasing the use-cases and tradeoffs between different caption quality metrics. We then use this language to describe our two-sided approach to caption quality improvement: (1) a gold-standard dataset creation methodology using stratified sampling and (2) a model alignment strategy encompassing context alignment and parameter-level finetuning using SFT. We demonstrate our methodology on open-source models, focusing on structured caption formats that enable better parsing and downstream utilization. We ultimately show that using a finetuned character detection model in an image captioning pipeline significantly improves holistic image-caption alignment quality. Our work addresses the growing need for high-quality"vegan"training data in enterprise AI development, providing practical solutions for teams seeking to improve caption-image alignment without relying on potentially copyright-protected web-scraped content.