Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions

2024-11-13

arXiv.org

AI Summary

Small-scale vision-language models (≤7B) suffer from hallucination and struggle to balance descriptive richness with factual accuracy during long visual-textual fine-tuning. To address this, we propose KnowAda, a knowledge-adaptive fine-tuning paradigm. Our key contributions are: (1) the first Decomposed Natural Language Inference (DNLI) framework for fine-grained hallucination quantification; (2) a confidence-based dynamic caption pruning and enhancement mechanism; and (3) knowledge distillation-guided data reweighting coupled with vision–language joint embedding alignment. Evaluated on multiple densely annotated datasets, KnowAda significantly outperforms baselines across automatic metrics—BLEU-4, SPICE, and CLIPScore—all improve substantially. Human evaluation shows a 23.6% increase in description accuracy and a 38.1% reduction in hallucination rate, demonstrating superior fidelity and expressiveness.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent research increasingly focuses on training vision-language models (VLMs) with long, detailed image captions. However, small-scale VLMs often struggle to balance the richness of these captions with the risk of hallucinating content during fine-tuning. In this paper, we explore how well VLMs adapt to such captions. To quantify caption quality, we propose Decomposed NLI (DNLI), an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. This fine-grained analysis reveals a critical balance between capturing descriptive details and preventing hallucinations. Our findings show that simply reducing caption complexity or employing standard data curation techniques does not effectively resolve this issue. To tackle this challenge, we introduce Knowledge Adapted (KnowAda) fine-tuning, a data-centric approach that automatically adapts training data with the model's existing knowledge and visual understanding. KnowAda minimizes hallucinations while preserving high descriptiveness. We validate this approach across several small-scale VLMs (up to 7B parameters) and dense caption datasets, demonstrating that KnowAda effectively balances hallucination reduction and descriptiveness. Our results show that KnowAda outperforms various baselines in both automatic metrics and human evaluations. We will release our code and models.

Problem

• Visual Language Models
• Detail Preservation
• Training Authenticity

Innovation

• KnowAda
• Automatic Data Adjustment
• Visual Language Model Improvement