RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities

2026-06-03

AI Summary

Existing disentangled representation learning methods struggle to scale beyond two modalities and fail to effectively model the complex interplay of shared and modality-specific factors in multimodal data. This work proposes RePercENT, a self-supervised framework featuring a plug-and-play multimodal architecture that operates directly on pre-extracted embeddings, enabling pairwise disentanglement across an arbitrary number of modalities without requiring joint pretraining. RePercENT is the first approach to overcome the bimodal limitation, offering scalable multimodal disentanglement. It introduces a jointly optimized objective that simultaneously extracts shared and private components, accompanied by theoretical guarantees of optimality. Experiments demonstrate that RePercENT successfully recovers disentangled representations across diverse modalities and tasks, achieving competitive performance while substantially reducing computational complexity.

Abstract

To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information. Learning disentangled representations is a principled way to identify these underlying shared and unique factors that are hidden in observational data. However, while multimodal disentanglement is a compelling paradigm, existing methods are largely confined to the two-modality regime due to its inherent scalability bottleneck. To address this, we propose RePercENT, a self-supervised framework designed to surpass these limitations and unlocks scalable pairwise disentanglement beyond two modalities. Through a multimodal `plug-and-play' architecture, our approach operates directly on pre-extracted embeddings, eliminating the need for extensive joint pre-training while making no assumptions regarding the underlying modalities or foundation model backbones. Moreover, we introduce a joint optimization objective for simultaneously deriving the shared and unique components, and provide formal theoretical guarantees that characterize the optimality of our solution. Across diverse modalities and tasks, RePercENT successfully recovers disentangled components while maintaining competitive performance and significantly reducing computational complexity.

Problem

• multimodal
• disentangled representation learning
• scalability
• cross-modal interactions
• modality-specific information

Innovation

• disentangled representation learning
• multimodal learning
• self-supervised learning
• scalable disentanglement
• plug-and-play architecture