reconstruction supervision

This work addresses the significant performance degradation of multimodal models under partial modality missingness, which stems from implicit modality preferences caused by imbalanced inter-modal learning during training. The study is the first to identify and quantify modality dominance relationships in the frequency domain and introduces a plug-and-play Multimodal Weight Allocation Module (MWAM). MWAM dynamically adjusts the contribution of each modality branch via a Frequency Ratio Metric (FRM), promoting balanced joint learning. This lightweight mechanism is highly generalizable and can be seamlessly integrated into both CNN and Vision Transformer (ViT) architectures. Extensive experiments demonstrate that it consistently enhances model robustness to missing modalities across diverse tasks and modality combinations, while also effectively boosting the performance of existing state-of-the-art methods.

Existing large multimodal models rely predominantly on text-only supervision, limiting their capacity to capture deep visual semantics and resulting in coarse-grained understanding and hallucination-prone behavior. To address this, we propose Reconstruction-based Visual Instruction Tuning (ROSS), a novel latent-space reconstruction paradigm oriented toward visual output: ROSS denoises and reconstructs image embeddings from the SigLIP encoder—bypassing RGB pixel-level regression—to enhance fine-grained visual perception and hallucination robustness. This is synergistically integrated with joint vision–language instruction tuning. ROSS consistently improves performance across diverse vision encoders and large language models. Notably, using only a single SigLIP encoder, ROSS achieves state-of-the-art results on multiple benchmarks, significantly outperforming multi-expert ensemble approaches while maintaining architectural simplicity and training efficiency.

This work addresses the limited synergy between understanding and generation modules in existing unified multimodal models, where their high degree of decoupling hinders mutual enhancement. To bridge this gap, the authors propose UNO, a comprehension-guided post-training framework that, for the first time, explicitly leverages understanding tasks—such as image captioning and visual regression—as supervisory signals during the generative process. By incorporating both semantic abstraction and structural detail objectives, UNO enables effective gradient flow from understanding to generation. Remarkably, with only lightweight post-training, the method achieves substantial performance gains in image generation and editing tasks, demonstrating that enhanced comprehension capabilities can significantly improve generative quality.

This work addresses the fundamental question of why multimodal contrastive pretraining (e.g., CLIP) enables zero-shot classification and cross-modal generation. Methodologically, we introduce the novel concept of *approximate sufficient statistics*, propose a joint hierarchical generative model for images and text, and integrate statistical inference, information theory, and Transformer approximation analysis to derive the first sample complexity upper bound for multimodal contrastive learning. We theoretically establish that contrastive representations are task-adaptive, and corroborate this via numerical simulations demonstrating strong generalization in zero-shot classification and cross-modal retrieval. Our primary contributions are: (i) uncovering the statistical essence underlying generalization in contrastive pretraining; (ii) quantifying its data efficiency via rigorous sample complexity bounds; and (iii) providing an interpretable, verifiable theoretical foundation for multimodal representation learning.

This work investigates whether synthetically generated images can serve as an effective auxiliary modality for text-centric tasks, thereby bridging the modality gap between unimodal language models and multimodal models. We propose the first systematic evaluation framework that integrates state-of-the-art text-to-image (T2I) models, prompt engineering techniques, and multimodal fusion architectures to enable collaborative learning between language models and synthetic images in text classification. Our key contributions are threefold: (1) the first empirical validation of synthetic images as a viable perceptual modality; (2) the establishment of a dedicated benchmark suite for this paradigm; and (3) the identification of semantic alignment, visual grounding capability, and generation fidelity as critical determinants of performance. Experiments demonstrate that synthetic images consistently enhance strong language model baselines—particularly on tasks with high visual grounding potential—confirming their effectiveness and practical promise.

Open-weight models pose challenges for component-level modification tasks (e.g., pruning or unlearning) when training data, loss functions, and gradient information are unavailable. Method: This paper proposes ModHiFi, an efficient, label-free, gradient-free importance estimation framework. Its core innovations are: (1) the Subset Fidelity metric, which quantifies global component importance via local reconstruction behavior—a first-of-its-kind formulation; and (2) a theoretical linkage between local and global reconstruction errors grounded in Lipschitz continuity, enabling fully unsupervised, data-free importance assessment. Results: ModHiFi-P achieves 11% higher speedup over state-of-the-art pruning methods on ImageNet. ModHiFi-U enables complete, zero-fine-tuning unlearning on CIFAR-10 and demonstrates strong generalization to Swin Transformers. Collectively, ModHiFi bridges a critical gap in model interpretability and editability under minimal supervision constraints.

Multimodal extension is often hindered by the high annotation cost of large-scale paired data, particularly in specialized domains such as medical imaging and molecular analysis. This work proposes TextME, a framework that, for the first time, maps diverse modalities—including images, audio, 3D, X-rays, and molecular data—into the embedding space of large language models using only textual descriptions, without any modality-paired supervision. By leveraging the geometric structure of pretrained contrastive encoders, TextME enables zero-shot cross-modal transfer purely through text-driven alignment. This approach establishes a novel paradigm for modality extension, achieving effective zero-shot retrieval across heterogeneous, unaligned modalities—such as audio-to-image or 3D-to-X-ray—while preserving the representational capacity of the pretrained encoders.

To address pervasive modality missingness and imbalance in clinical multimodal data, this paper proposes a robust multimodal fusion framework. It introduces learnable modality tokens to enable missingness-aware feature alignment; designs an enhanced modality dropout mechanism to explicitly model modality missing patterns; and incorporates cross-modal contrastive learning to improve generalization under single-modality inputs. The framework seamlessly integrates with state-of-the-art vision foundation models (e.g., CT-specific models) and supports joint modeling of visual data (e.g., medical images) and structured tabular data. Evaluated on large-scale real-world clinical datasets, the method significantly outperforms existing baselines—particularly under partial modality availability—while maintaining high accuracy, computational efficiency, and clinical applicability. Its robustness to heterogeneous missingness patterns, compatibility with modern vision architectures, and strong single-modality performance collectively advance practical deployment of multimodal learning in clinical settings.

This work addresses the limited generalization of existing machine learning models in cybersecurity tasks, which often rely on superficial features (i.e., shortcuts) in the data. To mitigate this, we propose the first two-stage multimodal contrastive learning framework tailored for cybersecurity, leveraging textual modalities—such as vulnerability descriptions—to guide threat classification in data-scarce payload modalities. By aligning their semantic embedding spaces, our approach enables effective cross-modal knowledge transfer. We construct a synthetic benchmark dataset using CVE entries and payloads generated by large language models, and validate our method on both a private large-scale dataset and public benchmarks. Experimental results demonstrate that our framework significantly alleviates shortcut learning, achieves superior generalization performance over existing baselines, and we publicly release the code and datasets to support further research.

This work addresses the unclear transferability mechanism between image understanding and generation tasks in existing unified multimodal models. It presents the first systematic investigation into cross-task transfer patterns between these two objectives and reveals that a shared Transformer backbone combined with a unified visual encoder enables stable knowledge transfer. Building on this insight, the study proposes a novel paradigm that enhances generative capabilities indirectly through training on understanding tasks, effectively mitigating distribution shift issues. The approach is validated on three critical competencies—counting, spatial reasoning, and text recognition/generation—demonstrating significant improvements in generation performance without compromising visual fidelity.