Structured Kernel Regression VAE: A Computationally Efficient Surrogate for GP-VAEs in ICA

2025-08-13

AI Summary

Addressing the challenge of jointly achieving interpretable latent disentanglement and computational efficiency in generative models, this paper proposes SKR-VAE: a variational autoencoder framework that replaces the Gaussian process (GP) prior with a Structured Kernel Regression (SKR) prior to enforce spatiotemporal disentanglement constraints on the latent space. By explicitly modeling structural dependencies among latent factors, SKR-VAE circumvents the O(N³) computational complexity inherent to GP inference. It retains ICA-based disentanglement performance comparable to GP-VAE while reducing both training and inference complexity to O(N²). Experiments on multiple benchmark datasets demonstrate that SKR-VAE significantly improves scalability for large-scale data processing, achieving strong disentanglement, high interpretability, and low computational overhead simultaneously.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The interpretability of generative models is considered a key factor in demonstrating their effectiveness and controllability. The generated data are believed to be determined by latent variables that are not directly observable. Therefore, disentangling, decoupling, decomposing, causal inference, or performing Independent Component Analysis (ICA) in the latent variable space helps uncover the independent factors that influence the attributes or features affecting the generated outputs, thereby enhancing the interpretability of generative models. As a generative model, Variational Autoencoders (VAEs) combine with variational Bayesian inference algorithms. Using VAEs, the inverse process of ICA can be equivalently framed as a variational inference process. In some studies, Gaussian processes (GPs) have been introduced as priors for each dimension of latent variables in VAEs, structuring and separating each dimension from temporal or spatial perspectives, and encouraging different dimensions to control various attributes of the generated data. However, GPs impose a significant computational burden, resulting in substantial resource consumption when handling large datasets. Essentially, GPs model different temporal or spatial structures through various kernel functions. Structuring the priors of latent variables via kernel functions-so that different kernel functions model the correlations among sequence points within different latent dimensions-is at the core of achieving disentanglement in VAEs. The proposed Structured Kernel Regression VAE (SKR-VAE) leverages this core idea in a more efficient way, avoiding the costly kernel matrix inversion required in GPs. This research demonstrates that, while maintaining ICA performance, SKR-VAE achieves greater computational efficiency and significantly reduced computational burden compared to GP-VAE.

Problem

• Enhancing interpretability of generative models via latent variable disentanglement
• Reducing computational burden of Gaussian processes in VAEs
• Efficiently structuring latent variable priors using kernel functions

Innovation

• SKR-VAE replaces GP-VAE for efficiency
• Kernel functions structure latent variable priors
• Avoids costly GP kernel matrix inversion