Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability

2025-01-02

AI Summary

This work addresses multi-level alignment between visual and linguistic representations in Large Vision-Language Models (LVLMs), systematically identifying three-tier semantic misalignments—object-, attribute-, and relation-level—and uncovering their root causes in data curation, model architecture, and inference dynamics. Method: We propose the first explainability-oriented LVLM alignment analysis framework, featuring a novel three-tier misalignment taxonomy. Leveraging integrated cross-modal diagnostics—including attribution analysis, feature visualization, and probing experiments—we combine theoretical modeling with empirical validation. Contribution/Results: We unify and comparatively analyze two major mitigation strategies—parameter freezing and fine-tuning—demonstrating their respective efficacy across alignment levels. Our findings enable the development of a standardized alignment evaluation protocol, advancing both theoretical understanding and practical design principles for robust, trustworthy multimodal models.

Technology Category

Computer Vision: Language and VisionMachine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Large language models for searchGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in processing both visual and textual information. However, the critical challenge of alignment between visual and linguistic representations is not fully understood. This survey presents a comprehensive examination of alignment and misalignment in LVLMs through an explainability lens. We first examine the fundamentals of alignment, exploring its representational and behavioral aspects, training methodologies, and theoretical foundations. We then analyze misalignment phenomena across three semantic levels: object, attribute, and relational misalignment. Our investigation reveals that misalignment emerges from challenges at multiple levels: the data level, the model level, and the inference level. We provide a comprehensive review of existing mitigation strategies, categorizing them into parameter-frozen and parameter-tuning approaches. Finally, we outline promising future research directions, emphasizing the need for standardized evaluation protocols and in-depth explainability studies.

Problem

• Visual-Language Alignment
• Multimodal Representation
• Cross-Modal Correspondence

Innovation

• Visual Language Models
• Alignment Issues
• Unified Evaluation Standards