multimodal fusion

Existing research lacks a systematic analysis of modality–language backbone integration mechanisms in multimodal large language models (MLLMs). Method: We systematically survey 125 MLLMs published between 2021 and 2025, proposing the first large-language-model-centric three-dimensional taxonomy—architectural integration, representation learning, and training paradigms—to unify cross-modal alignment pathways. Leveraging bibliometric analysis, architectural decoupling, and fine-grained modeling of embeddings and loss functions, we characterize evolutionary patterns in fusion granularity, joint representation design, and objective function development. Contribution/Results: This work fills a critical theoretical gap in structured analysis of MLLM fusion mechanisms, identifies shared bottlenecks—including misaligned modality-specific feature hierarchies and suboptimal cross-modal optimization objectives—and delivers a reusable theoretical framework and practical guidelines for developing robust, scalable multimodal foundation models.

This paper addresses key challenges in multimodal large language models (MLLMs)—namely limited scalability, insufficient robustness, and difficulties in cross-modal alignment—within vision-language tasks. To tackle these, it proposes a comprehensive technical taxonomy covering the full MLLM stack and introduces a tripartite evaluation framework centered on scalability, robustness, and ethical alignment. Methodologically, the work conducts a systematic comparative analysis of core techniques, including Transformer-based fusion architectures, visual encoders (e.g., ViT, CLIP), cross-modal alignment learning, instruction tuning, and multi-stage training. It further synthesizes structured best practices for modeling across text, image, video, and audio modalities. The study delivers 12 representative application cases, empirically validating its framework and guidelines. Collectively, this work advances trustworthy deployment and cross-domain adoption of MLLMs through rigorous architectural analysis, principled evaluation, and actionable implementation insights.

To address the challenge of efficiently extending multimodal capabilities of frozen large language models (LLMs), this paper proposes a dual-tower architecture that decouples visual and linguistic branches, injecting visual information solely via modality-specific adapters while keeping all LLM parameters frozen to preserve original language competence. Methodologically, it integrates cross-modal feature alignment with understanding-generation co-training. We identify three key empirical insights: (i) image-text understanding data substantially improves generation quality; (ii) denoised image preprocessing enhances overall performance; and (iii) feature alignment accelerates convergence—especially for smaller models. Experiments demonstrate that our approach consistently outperforms mainstream multimodal baselines on image–text bidirectional generation tasks. Notably, small-model convergence speed improves by over 40%, while generated image fidelity and text–image alignment are both significantly enhanced.

This study addresses insufficient deep integration of visual and linguistic representations. We propose FUSION-3B, a multimodal large language model featuring full-modality dynamic alignment. Methodologically, we introduce three novel components: (1) text-guided unified visual encoding, (2) context-aware recursive alignment decoding, and (3) dual-supervised semantic mapping loss—collectively transcending conventional late-fusion paradigms to enable pixel-level, question-level, and end-to-end cross-modal unified modeling. Despite its compact 3B parameter count, FUSION-3B achieves superior performance on most benchmarks using only 630 visual tokens—outperforming Cambrian-1 (8B) and Florence-VL (8B); even with only 300 visual tokens, it surpasses Cambrian-1 (8B), and it exceeds LLaVA-NeXT on over half of the evaluated benchmarks. To further support fine-grained vision–language alignment, we construct a language-driven synthetic QA dataset.

Large vision-language models (LVLMs) overly rely on deepest-layer visual features while neglecting complementary information across intermediate layers. Method: We propose an instruction-guided dynamic visual feature aggregation mechanism—the first of its kind to enable instruction-driven, adaptive selection, weighting, and cross-layer interaction of multi-depth visual features without increasing the number of visual tokens. Leveraging task-aware attention aggregation and instruction-conditioned gating, the method jointly optimizes fine-grained perception (via low-level features) and semantic understanding (via mid- to high-level features). Contribution/Results: Our approach achieves significant performance gains across 18 benchmarks spanning six diverse vision-language tasks, demonstrating both the effectiveness and strong generalizability of dynamic, hierarchical visual feature utilization.

This work challenges the prevailing view that image and text representations in vision-language models align only in deep layers. Inspired by DeepDream, the authors propose a synthesis method within an adapter-based architecture that extracts textual concept vectors layer by layer and optimizes corresponding images without auxiliary models or additional datasets. For the first time, this approach provides direct, concept-level evidence of cross-modal alignment starting from the very first layer: across seven network layers and hundreds of concepts, over 50% of images synthesized from the first layer already exhibit clear, salient visual features of target concepts—such as animals, activities, and seasons. This method offers an efficient and novel pathway toward enhancing the interpretability of vision-language models.

This work addresses the bottleneck in visual understanding of multimodal large language models (MLLMs) imposed by single-encoder representations, systematically exploring the design space of hybrid vision encoders. We propose three key techniques: (1) lightweight concatenative fusion of multi-resolution complementary vision encoders; (2) a Pre-Alignment mechanism that explicitly aligns vision tokens with language tokens in semantic space; and (3) an end-to-end joint training framework. Experiments demonstrate that our Eagle series models achieve new state-of-the-art performance across major benchmarks—including MMBench, OCRBench, and DocVQA—outperforming all open-source SOTA models. The approach significantly mitigates hallucination and delivers substantial gains on fine-grained tasks such as OCR and document parsing.

This work addresses the challenge of efficiently equipping vision-language models with continuously evolving domain-specific skills, a task hindered by the high cost of conventional fine-tuning. The authors propose injecting capabilities from domain-specialized large language models into vision-language models through model fusion, enabling cross-modal skill transfer without additional training data or substantial computational resources. The study presents the first systematic analysis of the method’s applicability, fusion strategies, and hyperparameter sensitivity, offering quantitative evaluations of techniques such as Task Arithmetic (TA) and DARE across heterogeneous architectures. Experimental results demonstrate strong performance in instruction-following and cross-lingual tasks, while revealing limitations in mathematical reasoning, thereby delineating the effective boundaries and critical tuning factors for cross-modal skill injection.

This work systematically investigates the core mechanisms of modality interaction in multimodal pretraining, focusing on knowledge transfer, synergistic effects, and fusion timing. Through experiments on both synthetic and large-scale real-world datasets, it provides the first empirical evidence of asymmetric cross-modal knowledge flow and demonstrates that data complexity governs whether modalities exhibit synergy or competition. The study further validates that early unified fusion consistently outperforms late alignment. Leveraging an architecture featuring shared attention and normalization layers with modality-specific feedforward components, the proposed approach is evaluated on a 13.5B mixture-of-experts model trained on 2 trillion tokens, confirming its effectiveness. Additionally, the paper introduces a highly efficient pretraining strategy that achieves strong generative performance using only 5% of the typical computational budget.

This study investigates the impact of fusion timing on the accuracy–latency trade-off in multimodal vision–language systems. We propose and systematically evaluate three fusion strategies—early, middle, and late—within a unified architecture combining BERT for language and lightweight visual backbones (MobileNetV2 or ViT) on the CMU MOSI dataset; inference latency is empirically measured on an NVIDIA Jetson Orin AGX edge platform. Results show that late fusion achieves the highest accuracy (12.3% lower MAE), while early fusion incurs the lowest latency (41.7% reduction on average), with fusion stage exhibiting a strong negative correlation between accuracy and latency. To our knowledge, this is the first work to quantitatively and systematically validate the critical influence of fusion location under a consistent experimental framework. Our findings provide reproducible architectural guidelines and empirical evidence for designing efficient multimodal models tailored to resource-constrained edge devices.

Multimodal learning faces critical challenges including difficulty in cross-source information fusion, poor robustness to modality missing, and vulnerability to adversarial attacks. To address these, we propose a robust multimodal representation learning framework. Methodologically, we design a contrastive learning–based cross-modal alignment mechanism with cross-attention, enabling unsupervised and self-supervised fusion; integrate AutoML-driven dynamic architecture search to enhance adaptability to incomplete inputs and adversarial perturbations; and establish a unified benchmarking framework for comprehensive evaluation. Our approach achieves significant performance gains on vision-language understanding and speech-text joint modeling tasks. Moreover, it introduces a reproducible, extensible evaluation standard system, advancing general-purpose multimodal representation paradigms. The framework demonstrates superior robustness under modality dropout and adversarial conditions while maintaining high accuracy across diverse multimodal benchmarks.

Current vision-language models (VLMs) exhibit strong perceptual capabilities but suffer from inherent limitations—including weak logical reasoning, inability to update knowledge without costly retraining, and poor interpretability. Method: This paper systematically surveys neural-symbolic integration approaches and proposes a fine-tuning-free collaborative reasoning framework: a pre-trained VLM serves as the neural frontend, tightly coupled with symbolic components—such as knowledge graphs and formal rule engines—to enable plug-and-play external knowledge injection and transparent, multi-step reasoning. Contribution/Results: We introduce the first taxonomy of neural-symbolic methods specifically designed for VLM enhancement, precisely delineating the applicability boundaries and bottlenecks of each technique in multimodal understanding. The framework provides a systematic solution to improve model interpretability, dynamic knowledge integration, and structured reasoning performance, advancing the state of explainable and adaptable multimodal AI.