InfoDPCCA: Information-Theoretic Dynamic Probabilistic Canonical Correlation Analysis

2025-06-10

AI Summary

High-dimensional bimodal time-series data (e.g., fMRI) pose challenges in jointly modeling shared and modality-specific latent structures while preserving discriminative and interpretable representations. Method: We propose Mutual Information-driven Dynamic Probabilistic Canonical Correlation Analysis (MI-DPCCA), a novel framework that jointly learns a shared latent representation—encoding only the mutual information between two sequences—and modality-specific latent components. For the first time, we integrate the information bottleneck principle into the dynamic probabilistic CCA framework, enforcing information constraints in the shared space via variational inference. Additionally, we introduce a two-stage training strategy and residual connections to enhance optimization stability and generative fidelity. Results: Evaluated on synthetic benchmarks and real fMRI datasets, MI-DPCCA significantly improves representation discriminability, interpretability, and downstream prediction performance, consistently outperforming DPCCA and other baselines across all metrics.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Extracting meaningful latent representations from high-dimensional sequential data is a crucial challenge in machine learning, with applications spanning natural science and engineering. We introduce InfoDPCCA, a dynamic probabilistic Canonical Correlation Analysis (CCA) framework designed to model two interdependent sequences of observations. InfoDPCCA leverages a novel information-theoretic objective to extract a shared latent representation that captures the mutual structure between the data streams and balances representation compression and predictive sufficiency while also learning separate latent components that encode information specific to each sequence. Unlike prior dynamic CCA models, such as DPCCA, our approach explicitly enforces the shared latent space to encode only the mutual information between the sequences, improving interpretability and robustness. We further introduce a two-step training scheme to bridge the gap between information-theoretic representation learning and generative modeling, along with a residual connection mechanism to enhance training stability. Through experiments on synthetic and medical fMRI data, we demonstrate that InfoDPCCA excels as a tool for representation learning. Code of InfoDPCCA is available at https://github.com/marcusstang/InfoDPCCA.

Problem

• Extract shared latent representations from high-dimensional sequential data
• Balance representation compression and predictive sufficiency in dynamic CCA
• Improve interpretability by isolating mutual information between sequences

Innovation

• Information-theoretic dynamic probabilistic CCA framework
• Two-step training scheme for representation learning
• Residual connection mechanism enhances training stability