multimodal fusion architecture

This study investigates the impact of fusion timing on the accuracy–latency trade-off in multimodal vision–language systems. We propose and systematically evaluate three fusion strategies—early, middle, and late—within a unified architecture combining BERT for language and lightweight visual backbones (MobileNetV2 or ViT) on the CMU MOSI dataset; inference latency is empirically measured on an NVIDIA Jetson Orin AGX edge platform. Results show that late fusion achieves the highest accuracy (12.3% lower MAE), while early fusion incurs the lowest latency (41.7% reduction on average), with fusion stage exhibiting a strong negative correlation between accuracy and latency. To our knowledge, this is the first work to quantitatively and systematically validate the critical influence of fusion location under a consistent experimental framework. Our findings provide reproducible architectural guidelines and empirical evidence for designing efficient multimodal models tailored to resource-constrained edge devices.

This work addresses the lack of principled understanding in existing literature regarding the choice between cross-attention and feature concatenation strategies for multimodal fusion, which has largely relied on empirical heuristics. Through controlled experiments and theoretical analysis, we demonstrate for the first time that feature alignment quality is the key determinant of fusion strategy performance: under pre-aligned features, concatenation consistently outperforms cross-attention by 4.1–5.1 percentage points across all data scales, with its advantage becoming more pronounced as alignment degrades. Building on this insight, we develop a theoretical decision framework grounded in sample complexity and validate our findings using features extracted from ResNet-18 and CLIP ViT-B/32 on controlled datasets.

This work addresses the limitations of existing approaches that predominantly rely on non-native late-fusion architectures and lack a systematic definition or unified framework for native multimodal modeling. The paper proposes a formal roadmap for Native Multimodal Modeling (NMM), introducing, for the first time, a rigorous formulation of “architectural nativeness.” Leveraging an input–output duality perspective, it categorizes models into three types: Multi-to-Text, Multi-to-Target, and Multi-to-Multi. A full-stack, industrial-grade NMM framework is developed, encompassing data governance, early/mid-stage fusion, end-to-end training, and deployment, all realized through a unified Transformer architecture that enables symbiotic cross-modal understanding and generation. This paradigm demonstrates superior performance and strong scalability across multiple tasks, offering a clear pathway toward truly native multimodal models.

Existing audio-visual fusion methods struggle to balance cross-modal dependency modeling with computational efficiency, limiting the scalability of multi-scale architectures. To address this challenge, this work proposes SNNergy, a novel framework that achieves hierarchical multi-scale cross-modal fusion with linear complexity for the first time. At its core lies the CMQKA mechanism, which leverages event-driven binary spiking operations to construct an efficient bidirectional Query-Key attention and integrates a learnable residual fusion strategy. Evaluated on benchmark datasets including CREMA-D, AVE, and UrbanSound8K-AV, the proposed method attains state-of-the-art performance, significantly outperforming existing approaches while demonstrating exceptional energy efficiency.

RGB-D multimodal fusion mechanisms have long suffered from poor interpretability, and the fundamental nature of cross-modal complementarity remains unclear. Method: This paper establishes the first interpretability analysis framework specifically targeting the fusion process, introducing a joint metric of semantic variance and feature similarity to systematically characterize cross-modal representation consistency, intra-modal evolutionary patterns, and collaborative optimization logic. Through cross-layer feature comparison and quantitative semantic analysis, we identify a prevalent imbalance between consistency and specificity in mainstream fusion strategies. Contribution/Results: We formalize a “specificity-driven inference under consistency constraints” principle that explicates cross-modal complementarity. Our framework provides both theoretical foundations and a verifiable evaluation paradigm for designing trustworthy, generalizable multimodal fusion models.

This work addresses the challenge that certain modalities may introduce interference under specific inputs in multimodal fusion, thereby degrading model performance. To mitigate this issue, the authors propose a plug-and-play pre-fusion calibration module that leverages cross-modal summary contrast to extract supportive and conflicting cues, generating instance-level and dimension-level modulation signals. These signals dynamically enhance beneficial features while suppressing misleading information. The method achieves, for the first time, fine-grained, conflict-aware modulation prior to fusion and is compatible with both sequential and convolutional architectures. It consistently improves performance across five benchmark tasks—including emotion understanding and action recognition—and demonstrates enhanced robustness and consistency under modality missingness and data perturbations.

This work challenges the prevailing assumption that complex multimodal architectures inherently outperform simpler approaches. To this end, we conduct a large-scale empirical study, systematically reproducing 19 state-of-the-art methods across nine benchmark datasets—spanning up to 23 modalities—under standardized evaluation protocols. We introduce SimBaMM, a lightweight Late-Fusion Transformer baseline, and rigorously assess it using automated hyperparameter optimization, missing-modality robustness evaluation, and statistical significance testing. Our results reveal that most sophisticated models fail to significantly outperform SimBaMM under fair, tuned conditions; in low-data regimes, they often underperform optimized unimodal baselines; and original publications frequently suffer from evaluation bias and irreproducibility. Contributions include: (1) the first reliability-aware evaluation framework for multimodal learning, (2) the SimBaMM baseline, and (3) a multimodal evaluation checklist—collectively establishing a reproducible, comparable benchmarking paradigm for the field.

This work addresses the challenge in multimodal sentiment analysis where modality-specific signal refinement and cross-modal interaction modeling often interfere with each other due to conflicting optimization objectives. To resolve this, the authors propose SeRIn, a novel architecture that decouples modality separation and cross-modal interaction into structured priors through a three-stage pipeline—separation, refinement, and integration—processing unimodal representations and cross-modal interactions via independent pathways before fusing them at the prediction stage. Notably, SeRIn adaptively adjusts modality weights without requiring explicit supervision. The method achieves state-of-the-art performance on the CH-SIMS and CMU-MOSEI benchmarks, yielding significant improvements across all evaluation metrics.

This work systematically investigates the core mechanisms of modality interaction in multimodal pretraining, focusing on knowledge transfer, synergistic effects, and fusion timing. Through experiments on both synthetic and large-scale real-world datasets, it provides the first empirical evidence of asymmetric cross-modal knowledge flow and demonstrates that data complexity governs whether modalities exhibit synergy or competition. The study further validates that early unified fusion consistently outperforms late alignment. Leveraging an architecture featuring shared attention and normalization layers with modality-specific feedforward components, the proposed approach is evaluated on a 13.5B mixture-of-experts model trained on 2 trillion tokens, confirming its effectiveness. Additionally, the paper introduces a highly efficient pretraining strategy that achieves strong generative performance using only 5% of the typical computational budget.

Current multimodal fusion evaluation is hindered by small-scale, narrow-domain, task-specific, and inconsistently standardized benchmarks, leading to poor model generalizability and incomparable results. To address this, we propose MMBench—the first large-scale, domain-adaptive multimodal fusion benchmark—integrating over 30 datasets, 15 modalities, and 20 predictive tasks across critical domains including healthcare, remote sensing, and industrial inspection. We design a unified cross-domain evaluation framework and an open-source automated pipeline supporting early-, late-, and hybrid-fusion paradigms. Our framework incorporates standardized preprocessing, cross-modal alignment, and domain-adaptation mechanisms. Extensive experiments establish multiple new state-of-the-art baselines, significantly improving model generalizability and reproducibility. MMBench provides a rigorous, open, and extensible evaluation infrastructure for advancing multimodal fusion research.