train multimodal models

This work systematically investigates the core mechanisms of modality interaction in multimodal pretraining, focusing on knowledge transfer, synergistic effects, and fusion timing. Through experiments on both synthetic and large-scale real-world datasets, it provides the first empirical evidence of asymmetric cross-modal knowledge flow and demonstrates that data complexity governs whether modalities exhibit synergy or competition. The study further validates that early unified fusion consistently outperforms late alignment. Leveraging an architecture featuring shared attention and normalization layers with modality-specific feedforward components, the proposed approach is evaluated on a 13.5B mixture-of-experts model trained on 2 trillion tokens, confirming its effectiveness. Additionally, the paper introduces a highly efficient pretraining strategy that achieves strong generative performance using only 5% of the typical computational budget.

Traditional multimodal learning relies on paired data to construct unified representations, yet leveraging **unpaired auxiliary modality data** to enhance target modality representation learning remains largely unexplored. This paper proposes the **Unimodal-agnostic Learning (UML) paradigm**, which implicitly captures structural correlations across modalities via parameter sharing and cross-modal alternating training—enabling knowledge transfer without explicit alignment. Under a linear generative assumption, theoretical analysis demonstrates that auxiliary modalities provide richer generative supervision signals. Methodologically, UML integrates contrastive learning with self-supervised strategies, ensuring strong generalizability. Empirically, incorporating unpaired auxiliary data—such as text, audio, or images—yields consistent and significant performance gains on diverse downstream tasks, including image classification and audio recognition. These results validate UML’s effectiveness, modality-agnostic design, and broad applicability across single-modality learning scenarios.

This paper systematically analyzes four core challenges in multimodal alignment and fusion—cross-modal misalignment, semantic modality gap, computational bottlenecks, and data noise/heterogeneity—that hinder model robustness, generalizability, and scalability. To address them, the authors propose a unified taxonomy covering 200+ studies, revealing an integrated “alignment–fusion” co-design paradigm. They further introduce three principled solution pathways: (1) noise-robust learning, (2) heterogeneous representation modeling, and (3) few-shot cross-modal transfer—unifying contrastive learning, cross-modal attention, latent-space alignment, graph neural networks, and differentiable architecture search. Extensive experiments on social media analysis, medical imaging, and sentiment recognition validate the efficacy of the framework. The work distills actionable design principles for scalable, robust, and generalizable multimodal learning, establishing a new benchmark for both theoretical research and industrial deployment.

Traditional multimodal approaches rely on modality-specific encoders and late fusion, limiting scalability and cross-modal generalization. This work proposes a unified paradigm that reformulates diverse multimodal tasks—including text, image, audio, and video processing—as “next-frame prediction,” with all inputs and outputs represented as serialized video frames, enabling end-to-end, single-model inference. It pioneers the task-redefinition strategy—previously confined to NLP—within multimodal learning, eliminating modality-specific design in favor of a fully modality-agnostic architecture. The framework integrates cross-modal tokenization, sequential frame representation, autoregressive modeling, and a shared Transformer decoder, supporting joint multi-task pretraining. Empirical evaluation demonstrates strong zero-shot and few-shot generalization across text-to-text, image-to-text, video-to-video, video-to-text, and audio-to-text tasks. Crucially, adaptation requires only lightweight task-specific heads, underscoring its flexibility and efficiency.

Existing surveys predominantly examine isolated components of multimodal pipelines and lack empirically grounded, pedagogically oriented integration frameworks for teaching and learning contexts. Method: This study introduces the first taxonomy and analytical framework covering five core modalities—natural language, video, sensor data, human-centered signals, and environmental logs—and proposes a novel “mid-fusion” paradigm for multimodal data integration. It further innovates by applying citation graph pruning to achieve structured, high-precision literature synthesis. Contribution/Results: Through systematic review, taxonomic modeling, and multimodal fusion design, we demonstrate that multimodal synergy enables detection of fine-grained learning behaviors imperceptible to unimodal analysis. While prediction accuracy remains largely unchanged, interpretability improves significantly, yielding deeper insights into learners’ cognitive-affective states and training outcomes.

This work addresses the limitations of existing approaches that predominantly rely on non-native late-fusion architectures and lack a systematic definition or unified framework for native multimodal modeling. The paper proposes a formal roadmap for Native Multimodal Modeling (NMM), introducing, for the first time, a rigorous formulation of “architectural nativeness.” Leveraging an input–output duality perspective, it categorizes models into three types: Multi-to-Text, Multi-to-Target, and Multi-to-Multi. A full-stack, industrial-grade NMM framework is developed, encompassing data governance, early/mid-stage fusion, end-to-end training, and deployment, all realized through a unified Transformer architecture that enables symbiotic cross-modal understanding and generation. This paradigm demonstrates superior performance and strong scalability across multiple tasks, offering a clear pathway toward truly native multimodal models.

Existing models are largely restricted to unimodal or bimodal architectures, limiting their capacity to efficiently integrate the rich diversity of visual modalities present in real-world scenarios. To address this, we propose a low-supervision, fully automated multimodal video data pipeline that enables programmable composition and joint learning across heterogeneous visual modalities—including RGB, depth, optical flow, and edge maps. Our key contributions are: (1) PHG-MAE, a lightweight multimodal self-supervised encoder (<1M parameters), which leverages pretrained expert models and knowledge distillation to achieve performance on par with 300M-parameter large models; and (2) seamless integration of off-the-shelf modules (e.g., DPT) to enable real-time semantic segmentation and near-real-time depth estimation from handheld or webcam video on commodity hardware. Extensive experiments demonstrate the pipeline’s efficiency, scalability, and strong generalization under resource-constrained conditions.

Conventional multimodal learning assumes explicit representation alignment is universally beneficial, yet its causal impact remains unverified. Method: This work systematically investigates the conditional effects of enforced alignment, proposing a controllable contrastive learning module that dynamically modulates alignment strength and establishes a quantitative relationship between alignment strength and inter-modal information redundancy—derived via information decomposition and synthetic data modeling. Contribution/Results: We demonstrate that alignment efficacy is not universal: strong alignment improves performance under high redundancy but harms modality-specific representation learning under low redundancy. To address this, we introduce a balanced mechanism that jointly preserves shared semantics and modality-specific characteristics. Empirical evaluation on both synthetic and real-world benchmarks confirms that this mechanism significantly enhances the generalization capability of unimodal encoders. Our core contribution is the formal establishment of “redundancy-dependent alignment”—a principled, interpretable, and tunable paradigm for multimodal representation learning.

Existing multimodal embedding models suffer from narrow modality coverage, training instability, and poor domain adaptability in industrial settings. To address these issues, this paper proposes a unified all-modal embedding foundation model. Methodologically, we design a multi-stage collaborative training framework integrating content-aware progressive learning and collaboration-aware recommendation enhancement; introduce a randomized specialization mechanism coupled with dataset-driven modality matching; and adopt a multi-tower architecture that jointly leverages large vision-language models (VLMs) and dual-path distillation—from sequences/IDs to items—to capture fine-grained user interests. The model significantly improves generalization and industrial robustness on cross-modal retrieval and recommendation tasks. Experiments demonstrate state-of-the-art performance across multiple benchmarks. In Douyin’s “Selected Content” scenario, it achieves +0.158% and +0.144% gains in 7-day and 14-day long-term user retention, respectively, and improves feed ranking AUC by 0.08%.

This work explores the design space of natively multimodal foundation models, addressing how to effectively integrate vision and language beyond conventional language modeling. Building upon the Transfusion framework, the authors propose a unified pretraining approach from scratch that jointly leverages next-token prediction and diffusion-based generation, augmented with a Representation Autoencoder (RAE) to unify visual representations for both understanding and generation. The study reveals the complementary nature and asymmetric scaling behavior of vision and language data—where vision benefits more substantially from increased data volume—and employs a Mixture-of-Experts (MoE) architecture to enable efficient modality specialization and model expansion. Experiments demonstrate that unified pretraining naturally induces world modeling capabilities and significantly enhances performance on downstream tasks, laying a foundation for truly integrated multimodal foundation models.