Disentangled representations via score-based variational autoencoders

2025-12-18

AI Summary

This work addresses three key challenges in unsupervised disentangled representation learning: (i) failure to recover true generative factors, (ii) lack of semantic interpretability, and (iii) curved latent trajectories in video sequences. To this end, we propose SAMI—a novel framework that rigorously unifies the objective functions of diffusion models and VAEs for the first time, optimizing a multi-scale evidence lower bound via score-guided latent-space diffusion. SAMI requires no supervision and automatically identifies semantically meaningful axes in the latent space. On synthetic data, it precisely recovers ground-truth generative factors; on natural images and videos, it yields representations with higher disentanglement, clearer semantic structure, and significantly straighter latent trajectories. Moreover, SAMI enables zero-shot extraction of interpretable representations from pretrained diffusion models, achieving effective knowledge transfer with only minimal fine-tuning.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present the Score-based Autoencoder for Multiscale Inference (SAMI), a method for unsupervised representation learning that combines the theoretical frameworks of diffusion models and VAEs. By unifying their respective evidence lower bounds, SAMI formulates a principled objective that learns representations through score-based guidance of the underlying diffusion process. The resulting representations automatically capture meaningful structure in the data: it recovers ground truth generative factors in our synthetic dataset, learns factorized, semantic latent dimensions from complex natural images, and encodes video sequences into latent trajectories that are straighter than those of alternative encoders, despite training exclusively on static images. Furthermore, SAMI can extract useful representations from pre-trained diffusion models with minimal additional training. Finally, the explicitly probabilistic formulation provides new ways to identify semantically meaningful axes in the absence of supervised labels, and its mathematical exactness allows us to make formal statements about the nature of the learned representation. Overall, these results indicate that implicit structural information in diffusion models can be made explicit and interpretable through synergistic combination with a variational autoencoder.

Problem

• Unsupervised learning of disentangled representations from data
• Combining diffusion models with variational autoencoders for structure
• Extracting interpretable semantic axes without supervised labels

Innovation

• Combines diffusion models and VAEs for representation learning
• Uses score-based guidance to learn disentangled latent dimensions
• Extracts interpretable representations from pre-trained diffusion models