design dual-stream networks

Cross-modal learning suffers from modality dominance, redundant coupling, and spurious correlations, leading to poor generalization, weak interpretability, and insufficient robustness to noise or missing modalities. To address these issues, we propose the Dual-Stream Residual Semantic Disentanglement (DRSD) framework, which explicitly separates modality-specific representations from shared semantic representations via residual decomposition and orthogonal regularization—thereby mitigating cross-modal redundancy and enhancing weak-signal modeling. DRSD integrates a dual-stream architecture, a residual semantic alignment head, contrastive-regressive joint optimization, and covariance-based regularization. Evaluated on two large-scale educational benchmarks, DRSD significantly outperforms unimodal, early-fusion, late-fusion, and co-attention baselines in both next-step and final outcome prediction. It achieves superior generalization, robustness to missing modalities, and enhanced interpretability through disentangled, semantically grounded representations.

This study addresses the unclear mechanisms underlying cross-layer fusion of visual and textual information in multimodal large language models. We propose an architecture-aware diagnostic framework that systematically compares concatenation-based and native multimodal architectures. Through alignment decoupling, attention entropy analysis, intrinsic dimensionality estimation, causal intervention, and vision-specific Centered Kernel Alignment (CKA), we reveal how feature spaces are reorganized under different architectural paradigms. Our findings indicate that concatenation-based models exhibit a text-dominant fusion trajectory, whereas native models achieve early-stage vision-language co-adaptation. This work elucidates the distinct multimodal fusion mechanisms inherent to these two architectural paradigms, providing a principled theoretical foundation for future model design.

This work addresses the lack of principled understanding in existing literature regarding the choice between cross-attention and feature concatenation strategies for multimodal fusion, which has largely relied on empirical heuristics. Through controlled experiments and theoretical analysis, we demonstrate for the first time that feature alignment quality is the key determinant of fusion strategy performance: under pre-aligned features, concatenation consistently outperforms cross-attention by 4.1–5.1 percentage points across all data scales, with its advantage becoming more pronounced as alignment degrades. Building on this insight, we develop a theoretical decision framework grounded in sample complexity and validate our findings using features extracted from ResNet-18 and CLIP ViT-B/32 on controlled datasets.

This study investigates the impact of fusion timing on the accuracy–latency trade-off in multimodal vision–language systems. We propose and systematically evaluate three fusion strategies—early, middle, and late—within a unified architecture combining BERT for language and lightweight visual backbones (MobileNetV2 or ViT) on the CMU MOSI dataset; inference latency is empirically measured on an NVIDIA Jetson Orin AGX edge platform. Results show that late fusion achieves the highest accuracy (12.3% lower MAE), while early fusion incurs the lowest latency (41.7% reduction on average), with fusion stage exhibiting a strong negative correlation between accuracy and latency. To our knowledge, this is the first work to quantitatively and systematically validate the critical influence of fusion location under a consistent experimental framework. Our findings provide reproducible architectural guidelines and empirical evidence for designing efficient multimodal models tailored to resource-constrained edge devices.

This work identifies a fundamental conflict in unified multimodal models: understanding tasks require progressively strengthened cross-modal alignment across network depth to build semantics, whereas generation tasks necessitate shallow alignment and deep disentanglement to preserve spatial fidelity. To resolve this, we propose UniFork—a Y-shaped architecture featuring a shared shallow encoder for generic cross-modal representation learning and task-specific deep branches that explicitly decouple alignment dynamics per task. Guided by modality alignment behavior analysis and task-aware depth-wise disentanglement design, we validate UniFork’s effectiveness via multi-stage ablation studies. On diverse understanding and generation benchmarks, UniFork surpasses fully shared Transformers and matches or exceeds single-task expert models—achieving bidirectional state-of-the-art performance within a single unified architecture.

This work addresses the limitations of existing approaches that predominantly rely on non-native late-fusion architectures and lack a systematic definition or unified framework for native multimodal modeling. The paper proposes a formal roadmap for Native Multimodal Modeling (NMM), introducing, for the first time, a rigorous formulation of “architectural nativeness.” Leveraging an input–output duality perspective, it categorizes models into three types: Multi-to-Text, Multi-to-Target, and Multi-to-Multi. A full-stack, industrial-grade NMM framework is developed, encompassing data governance, early/mid-stage fusion, end-to-end training, and deployment, all realized through a unified Transformer architecture that enables symbiotic cross-modal understanding and generation. This paradigm demonstrates superior performance and strong scalability across multiple tasks, offering a clear pathway toward truly native multimodal models.

This work addresses the challenge that certain modalities may introduce interference under specific inputs in multimodal fusion, thereby degrading model performance. To mitigate this issue, the authors propose a plug-and-play pre-fusion calibration module that leverages cross-modal summary contrast to extract supportive and conflicting cues, generating instance-level and dimension-level modulation signals. These signals dynamically enhance beneficial features while suppressing misleading information. The method achieves, for the first time, fine-grained, conflict-aware modulation prior to fusion and is compatible with both sequential and convolutional architectures. It consistently improves performance across five benchmark tasks—including emotion understanding and action recognition—and demonstrates enhanced robustness and consistency under modality missingness and data perturbations.

This work systematically investigates the core mechanisms of modality interaction in multimodal pretraining, focusing on knowledge transfer, synergistic effects, and fusion timing. Through experiments on both synthetic and large-scale real-world datasets, it provides the first empirical evidence of asymmetric cross-modal knowledge flow and demonstrates that data complexity governs whether modalities exhibit synergy or competition. The study further validates that early unified fusion consistently outperforms late alignment. Leveraging an architecture featuring shared attention and normalization layers with modality-specific feedforward components, the proposed approach is evaluated on a 13.5B mixture-of-experts model trained on 2 trillion tokens, confirming its effectiveness. Additionally, the paper introduces a highly efficient pretraining strategy that achieves strong generative performance using only 5% of the typical computational budget.

研究解决了多模态分割学习中带宽分配问题，通过基于Shapley贡献值动态调整每种模态的数据保留率，提高了模型准确性和压缩效率。

Neural networks are often treated as monolithic black boxes, lacking maintainability and systematic verifiability. This work formally defines the notion of “decomposability” for neural networks, establishing semantic contracts between the original model and its components based on decision boundary semantics preservation. To realize semantic-aware, verification-driven decomposition, the authors propose the SAVED framework, which integrates boundary-aware counterexample mining, low logical margin input analysis, probabilistic coverage evaluation, and structure-aware pruning. Experiments across CNNs, language Transformers, and vision Transformers reveal a fundamental architectural disparity: language models tend to satisfy decomposability more readily, whereas vision models frequently violate this property, highlighting intrinsic differences in their structural amenability to semantic decomposition.