Structured Latent Modeling for Supervised Multimodal Information Decomposition

2026-09-28

AI Summary

This study addresses the challenge of effectively disentangling shared, modality-specific, and cross-modal information from continuous representations in multimodal prediction. We propose an explicit disentanglement framework based on latent variables, which decomposes the joint distribution by leveraging invertible normalizing flows and low-rank latent variable models. Furthermore, this approach integrates intermediate-layer contrastive learning with masked autoencoding objectives to achieve unified representation learning guided by structured likelihoods. By analyzing continuous multimodal interaction mechanisms from a latent variable perspective, our method yields robust multimodal information disentanglement and significantly improves predictive performance across multiple benchmarks.

Abstract

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

Problem

• Multimodal prediction
• Information decomposition
• Latent variable model
• Representation learning
• Cross-modal dependencies

Innovation

• Multimodal Information Decomposition
• Normalizing Flows
• Low-rank Latent Variable Model
• Contrastive Learning
• Structured Likelihood