Rethinking VAE: From Continuous to Discrete Representations Without Probabilistic Assumptions

2025-07-23

AI Summary

This paper addresses the conceptual gap between variational autoencoders (VAEs) and vector-quantized VAEs (VQ-VAEs) in modeling continuous versus discrete latent representations. Methodologically, it proposes a novel autoencoder framework that eliminates both the KL divergence term and the reparameterization trick; instead, it explicitly enforces latent space compactness via learnable clustering centers and employs multi-vector outputs to jointly support continuous interpolation and discrete reconstruction. Key contributions include: (1) uncovering an intrinsic relationship between autoencoder generative fidelity and latent space compactness; (2) establishing a deterministic transition from VAEs to VQ-VAEs without relying on probabilistic assumptions; and (3) empirically validating smooth interpolation and stable reconstruction on MNIST, CelebA, and FashionMNIST. Experiments further reveal that naively increasing the number of output vectors leads to model degradation—manifesting as localized, patchwise discrete encoding—highlighting the critical role of architectural design.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

This paper explores the generative capabilities of Autoencoders (AEs) and establishes connections between Variational Autoencoders (VAEs) and Vector Quantized-Variational Autoencoders (VQ-VAEs) through a reformulated training framework. We demonstrate that AEs exhibit generative potential via latent space interpolation and perturbation, albeit limited by undefined regions in the encoding space. To address this, we propose a new VAE-like training method that introduces clustering centers to enhance data compactness and ensure well-defined latent spaces without relying on traditional KL divergence or reparameterization techniques. Experimental results on MNIST, CelebA, and FashionMNIST datasets show smooth interpolative transitions, though blurriness persists. Extending this approach to multiple learnable vectors, we observe a natural progression toward a VQ-VAE-like model in continuous space. However, when the encoder outputs multiple vectors, the model degenerates into a discrete Autoencoder (VQ-AE), which combines image fragments without learning semantic representations. Our findings highlight the critical role of encoding space compactness and dispersion in generative modeling and provide insights into the intrinsic connections between VAEs and VQ-VAEs, offering a new perspective on their design and limitations.

Problem

• Enhancing AE generative ability via latent space clustering
• Exploring VAE-VQVAE connections without probabilistic assumptions
• Addressing blurriness in interpolations via compact latent spaces

Innovation

• Proposes VAE-like training with clustering centers
• Enhances data compactness without KL divergence
• Connects VAEs and VQ-VAEs via reformulated framework