MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

2026-07-18

AI Summary

This work addresses the challenges of scarce large-scale aligned data and insufficient interpretability and controllability in multimodal learning, which arise from the entanglement of shared and modality-specific information. To this end, we propose an efficient multimodal learning framework based on Low-Rank Representation Finetuning (LoReFT). Our approach extends low-rank adaptation to multimodal settings for the first time, leveraging pretrained unimodal models to explicitly disentangle and reveal the structural distribution of cross-modal shared and modality-unique information through an interpretable projection subspace. Experimental results demonstrate that the proposed framework effectively generates representations that support multimodal prediction on both synthetic and real-world benchmarks, significantly enhancing model interpretability and controllability.

Abstract

Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities. However, training multimodal models faces two main obstacles. First, collecting large-scale, well-aligned paired multimodal datasets is often impractical, making end-to-end multimodal training difficult. Second, existing multimodal representations frequently entangle information shared across modalities with modality-specific information, hindering interpretability and control. We introduce MultiLoReFT, an efficient and scalable low-rank representation fine-tuning framework for multimodal learning with pretrained unimodal models. MultiLoReFT extends low-rank adaptation to the multimodal setting and learns interpretable projection subspaces that decouple shared and modality-specific information. Across simulated and real-world benchmarks, it produces representations that support multimodal prediction while explicitly revealing how shared and modality-specific information is distributed across modalities.

Problem

• multimodal learning
• shared information
• modality-specific information
• data alignment
• representation entanglement

Innovation

• MultiLoReFT
• low-rank adaptation
• multimodal learning
• information disentanglement
• representation fine-tuning