fuse heterogeneous representations

To address insufficient cross-modal fusion and excessive parameter overhead in multimodal model fine-tuning, this paper proposes the Heterogeneous Mixture-of-Experts Adapter (Heterogeneous MoE Adapter), the first to integrate heterogeneous Mixture of Experts into parameter-efficient fine-tuning (PEFT) for multimodal models. Under frozen backbone parameters, our method introduces modality-aware low-rank affine experts and a dynamic gating mechanism to enable efficient cross-modal collaboration and deep semantic fusion. Evaluated on eight vision-audio and vision-text downstream tasks, it achieves state-of-the-art performance while tuning only 5–8% of the total parameters—significantly outperforming unimodal PEFT baselines. This demonstrates the critical role of heterogeneous expert modeling in enhancing multimodal representation learning.

This work addresses the limitations of existing approaches that predominantly rely on non-native late-fusion architectures and lack a systematic definition or unified framework for native multimodal modeling. The paper proposes a formal roadmap for Native Multimodal Modeling (NMM), introducing, for the first time, a rigorous formulation of “architectural nativeness.” Leveraging an input–output duality perspective, it categorizes models into three types: Multi-to-Text, Multi-to-Target, and Multi-to-Multi. A full-stack, industrial-grade NMM framework is developed, encompassing data governance, early/mid-stage fusion, end-to-end training, and deployment, all realized through a unified Transformer architecture that enables symbiotic cross-modal understanding and generation. This paradigm demonstrates superior performance and strong scalability across multiple tasks, offering a clear pathway toward truly native multimodal models.

To address the challenges of online learning under dynamic concept drift and inter-stream heterogeneity in heterogeneous multi-data-stream settings, this paper proposes the CAMEL framework. Methodologically, CAMEL (1) assigns each stream an independent feature extractor and task head to explicitly model stream-wise heterogeneity; (2) introduces a collaborative mixture-of-experts mechanism augmented with multi-head attention for context-aware knowledge coordination and targeted transfer; and (3) incorporates an autonomous expert tuning strategy that enables dynamic expert instantiation, incremental updating, and pruning—thereby jointly enhancing adaptability to concept drift and mitigating catastrophic forgetting. Extensive experiments across diverse multi-stream benchmarks demonstrate that CAMEL consistently outperforms state-of-the-art methods, achieving significant improvements in generalization, robustness, and continual learning efficiency.

This study investigates the impact of fusion timing on the accuracy–latency trade-off in multimodal vision–language systems. We propose and systematically evaluate three fusion strategies—early, middle, and late—within a unified architecture combining BERT for language and lightweight visual backbones (MobileNetV2 or ViT) on the CMU MOSI dataset; inference latency is empirically measured on an NVIDIA Jetson Orin AGX edge platform. Results show that late fusion achieves the highest accuracy (12.3% lower MAE), while early fusion incurs the lowest latency (41.7% reduction on average), with fusion stage exhibiting a strong negative correlation between accuracy and latency. To our knowledge, this is the first work to quantitatively and systematically validate the critical influence of fusion location under a consistent experimental framework. Our findings provide reproducible architectural guidelines and empirical evidence for designing efficient multimodal models tailored to resource-constrained edge devices.

To address communication overhead, computational redundancy, and excessive memory consumption in training Mixture-of-Experts (MoE) models on heterogeneous hardware, this paper proposes the first hardware-aware expert assignment framework. Our method introduces three key innovations: (1) expert-specific operators enabling zero-redundancy in-place computation; (2) a dual-centered (data- and model-driven) adaptive parallelism configuration mechanism; and (3) device-level pipelined shared caching. Evaluated under realistic heterogeneous cluster settings, the framework preserves model accuracy while reducing memory footprint by 10–48% and accelerating training throughput by 0.5–4.3×. Consequently, end-to-end training latency is significantly lowered. This work provides a systematic solution for efficient large-scale MoE deployment across heterogeneous infrastructure.

This work addresses the challenge of effectively fusing heterogeneous language models, which is hindered by architectural disparities, misaligned parameter spaces, and conflicting knowledge. To overcome these limitations, the authors propose HeteroFusion, a novel approach that replaces conventional parameter matching with topological alignment of functional modules and incorporates a conflict-aware denoising mechanism to suppress incompatible signals. HeteroFusion enables, for the first time, efficient fusion across distinct model families such as Llama, Qwen, and Mistral. By integrating adapter-based preservation with structured parameter update strategies, the method substantially enhances fusion stability and cross-family generalization. Experimental results demonstrate that HeteroFusion consistently outperforms existing baselines in heterogeneous model fusion, multi-source ensemble tasks, and robustness to noisy inputs.

This work systematically investigates the core mechanisms of modality interaction in multimodal pretraining, focusing on knowledge transfer, synergistic effects, and fusion timing. Through experiments on both synthetic and large-scale real-world datasets, it provides the first empirical evidence of asymmetric cross-modal knowledge flow and demonstrates that data complexity governs whether modalities exhibit synergy or competition. The study further validates that early unified fusion consistently outperforms late alignment. Leveraging an architecture featuring shared attention and normalization layers with modality-specific feedforward components, the proposed approach is evaluated on a 13.5B mixture-of-experts model trained on 2 trillion tokens, confirming its effectiveness. Additionally, the paper introduces a highly efficient pretraining strategy that achieves strong generative performance using only 5% of the typical computational budget.

This work proposes a multi-stream parallel Transformer architecture that overcomes the limitations of conventional language models, which rely on a single sequential pipeline and suffer from high latency and tight coupling among input processing, reasoning, and output generation. By decoupling and synchronously handling these three stages at the data-driven level for the first time, the proposed framework enables concurrent execution while supporting multi-channel causal modeling and synchronized inference through instruction fine-tuning. This approach substantially enhances real-time responsiveness in interactive scenarios and improves system security, monitorability, and overall efficiency by enforcing clear separation of responsibilities across processing streams.

This work addresses the challenge of mismatched computational demands between vision encoders and large language models (LLMs) under long-context scenarios in multimodal large language model training, where conventional LLM-centric parallelization strategies fall short. The authors propose a heterogeneous parallel framework that decouples the encoder and LLM for the first time, enabling module-level independent parallelism and flexible device placement—either co-located or non-co-located. A boundary communicator ensures tensor semantic consistency across modules, while an extended scheduling mechanism supports diverse parallelism combinations, including tensor (TP), context (CP), pipeline (PP), data (DP), and expert (EP) parallelism. Experiments demonstrate that the proposed approach achieves up to 49.3% higher TFLOPS/GPU in co-located configurations and improves token throughput by 13.0% and TFLOPS/GPU by 9.6% in non-co-located settings, all while maintaining convergence performance on par with baseline methods.

This study addresses the inefficiency of manual design and the difficulty of systematically exploring architectural combinations in heterogeneous Mixture-of-Experts (MoE) models. The authors propose the first deterministic code-assembled generator tailored for heterogeneous MoE, establishing an automated search pipeline that efficiently generates and evaluates four-expert heterogeneous MoE architectures on the LEMUR dataset. Their approach integrates a convolutional gating network, temperature scaling, mixup augmentation, and cosine annealing learning rate scheduling. To mitigate search bias introduced by lexicographic enumeration, they employ stratified random sampling to enhance uniformity in architectural space coverage. Within 28 days, the pipeline produced 4,463 candidate models, successfully evaluating 1,021 of them. The combination of ShuffleNet and MobileNetV3 emerged as the top-performing architecture (mean accuracy 0.632), while FractalNet and MNASNet were identified as consistently underperforming families.