late fusion

Conventional multimodal learning assumes explicit representation alignment is universally beneficial, yet its causal impact remains unverified. Method: This work systematically investigates the conditional effects of enforced alignment, proposing a controllable contrastive learning module that dynamically modulates alignment strength and establishes a quantitative relationship between alignment strength and inter-modal information redundancy—derived via information decomposition and synthetic data modeling. Contribution/Results: We demonstrate that alignment efficacy is not universal: strong alignment improves performance under high redundancy but harms modality-specific representation learning under low redundancy. To address this, we introduce a balanced mechanism that jointly preserves shared semantics and modality-specific characteristics. Empirical evaluation on both synthetic and real-world benchmarks confirms that this mechanism significantly enhances the generalization capability of unimodal encoders. Our core contribution is the formal establishment of “redundancy-dependent alignment”—a principled, interpretable, and tunable paradigm for multimodal representation learning.

Multimodal learning faces critical challenges including difficulty in cross-source information fusion, poor robustness to modality missing, and vulnerability to adversarial attacks. To address these, we propose a robust multimodal representation learning framework. Methodologically, we design a contrastive learning–based cross-modal alignment mechanism with cross-attention, enabling unsupervised and self-supervised fusion; integrate AutoML-driven dynamic architecture search to enhance adaptability to incomplete inputs and adversarial perturbations; and establish a unified benchmarking framework for comprehensive evaluation. Our approach achieves significant performance gains on vision-language understanding and speech-text joint modeling tasks. Moreover, it introduces a reproducible, extensible evaluation standard system, advancing general-purpose multimodal representation paradigms. The framework demonstrates superior robustness under modality dropout and adversarial conditions while maintaining high accuracy across diverse multimodal benchmarks.

This paper addresses the substantial storage and computational overheads induced by high-dimensional multimodal embeddings, as well as the challenge of preserving cross-modal semantic consistency. We propose a fine-tuning-free semantic compression method. Its core insight is that smaller inter-modal semantic gaps correlate with higher compressibility; thus, we replace original high-dimensional embeddings with shared semantic representatives—namely, cluster centroids derived from each modality’s embedding space. Leveraging pretrained encoders and multimodal alignment techniques, our approach constructs a unified semantic center to yield compact, cross-modal representations. Extensive evaluation across multiple large-scale multimodal benchmarks demonstrates: (i) significant memory reduction (average compression rate >60%), (ii) zero downstream performance degradation, (iii) modality-agnostic applicability, and (iv) high deployment efficiency.

This study investigates whether spatial alignment in multimodal representation learning degrades modality-specific information—particularly in remote sensing fusion of heterogeneous sources (e.g., optical and SAR). We first establish a theoretical analysis framework revealing how alignment operations inherently erode modality-unique semantic content. To address this, we propose a self-supervised contrastive learning paradigm that jointly optimizes semantic alignment and modality fidelity. Extensive experiments on real-world remote sensing datasets demonstrate that aggressive spatial alignment improves cross-modal consistency but substantially compromises modality-discriminative feature representation. Our method preserves alignment performance while boosting modality-specific representation capability by 12.7% (average improvement). The work provides an interpretable trade-off principle between alignment and specificity for multimodal remote sensing fusion and releases open-source code and a benchmark dataset.

This study challenges the implicit “more modalities, better performance” assumption in multimodal deep learning (MDL) for computational pathology, focusing on survival time prediction for prostate cancer biochemical recurrence. We address the problem that integrating low-performing modalities—such as histopathology images, MRI, and clinical variables—can introduce noise and degrade predictive accuracy. To mitigate this, we propose a performance-guided multimodal fusion strategy: only modalities demonstrating strong independent prognostic value in survival analysis are selected for fusion. Experimental results show that selective fusion significantly improves both the concordance index (C-index) and Brier score, whereas inclusion of low-performing modalities consistently harms performance. To our knowledge, this is the first systematic investigation validating the critical impact of modality quality on MDL efficacy in survival prediction. Our work establishes modality selection—not merely fusion—as a fundamental step for enhancing robustness and reliability in multimodal survival modeling.

This work addresses the challenge in multimodal sentiment analysis where modality-specific signal refinement and cross-modal interaction modeling often interfere with each other due to conflicting optimization objectives. To resolve this, the authors propose SeRIn, a novel architecture that decouples modality separation and cross-modal interaction into structured priors through a three-stage pipeline—separation, refinement, and integration—processing unimodal representations and cross-modal interactions via independent pathways before fusing them at the prediction stage. Notably, SeRIn adaptively adjusts modality weights without requiring explicit supervision. The method achieves state-of-the-art performance on the CH-SIMS and CMU-MOSEI benchmarks, yielding significant improvements across all evaluation metrics.

Existing performance-degradation-based methods for modality contribution assessment struggle to disentangle a modality’s unique information content from its synergistic interaction effects—particularly under cross-attention architectures, where they face fundamental limitations. To address this, we propose the first representation-level quantification framework grounded in Partial Information Decomposition (PID), which rigorously decomposes multimodal representations into unique, redundant, and synergistic information components. Our method integrates PID theory with the Iterative Proportional Fitting Procedure (IPFP), enabling layer-wise and cross-dataset contribution inference without retraining. Experiments demonstrate that our framework substantially enhances both the interpretability and accuracy of contribution analysis. It provides a principled, fine-grained separation of each modality’s independent and interactive contributions, establishing a new paradigm for multimodal model diagnosis and architecture design.

This work addresses the lack of principled understanding in existing literature regarding the choice between cross-attention and feature concatenation strategies for multimodal fusion, which has largely relied on empirical heuristics. Through controlled experiments and theoretical analysis, we demonstrate for the first time that feature alignment quality is the key determinant of fusion strategy performance: under pre-aligned features, concatenation consistently outperforms cross-attention by 4.1–5.1 percentage points across all data scales, with its advantage becoming more pronounced as alignment degrades. Building on this insight, we develop a theoretical decision framework grounded in sample complexity and validate our findings using features extracted from ResNet-18 and CLIP ViT-B/32 on controlled datasets.

Existing Omni-MLLMs suffer from perceptual fragility due to their static fusion architectures, often underperforming single-modality baselines in multimodal joint reasoning. This work proposes the Chain of Modality (CoM) framework, which for the first time enables dynamic switching of multimodal fusion topologies, adaptively selecting among parallel, sequential, or interleaved input structures based on task demands. CoM incorporates dual cognitive pathways—intuitive and deliberative decision-making—to better align model behavior with task requirements. Requiring either no training or only data-efficient supervised fine-tuning, the method leverages dynamic routing and attention topology modulation to consistently and significantly outperform existing static fusion approaches across multiple benchmarks.

This paper addresses the challenge of jointly modeling strong pairwise alignment and higher-order (e.g., XOR-type) inter-modal dependencies in multimodal joint representation learning. To this end, we propose ConFu, a contrastive fusion framework that jointly optimizes unimodal and fused multimodal representations within a unified embedding space. ConFu introduces, for the first time, a fused-modal contrastive loss that explicitly captures higher-order interactions and enables both one-to-one bidirectional and two-to-one cross-modal retrieval. By extending the contrastive learning objective and co-optimizing multimodal fusion encoders with the joint embedding space, ConFu achieves significant improvements over state-of-the-art methods on synthetic and real-world benchmarks—including MM-IMDB and Clotho—across cross-modal retrieval and classification tasks. Moreover, the framework exhibits strong computational scalability.