design variational autoencoders

This paper addresses the conceptual gap between variational autoencoders (VAEs) and vector-quantized VAEs (VQ-VAEs) in modeling continuous versus discrete latent representations. Methodologically, it proposes a novel autoencoder framework that eliminates both the KL divergence term and the reparameterization trick; instead, it explicitly enforces latent space compactness via learnable clustering centers and employs multi-vector outputs to jointly support continuous interpolation and discrete reconstruction. Key contributions include: (1) uncovering an intrinsic relationship between autoencoder generative fidelity and latent space compactness; (2) establishing a deterministic transition from VAEs to VQ-VAEs without relying on probabilistic assumptions; and (3) empirically validating smooth interpolation and stable reconstruction on MNIST, CelebA, and FashionMNIST. Experiments further reveal that naively increasing the number of output vectors leads to model degradation—manifesting as localized, patchwise discrete encoding—highlighting the critical role of architectural design.

Traditional variational autoencoders (VAEs) are constrained by a standard isotropic Gaussian prior, which often fails to accurately capture the complex latent structure of real-world data, leading to suboptimal generation quality and reconstruction fidelity. To address this limitation, this work proposes the X-VAE framework, which innovatively constructs a data-adaptive Gaussian mixture prior by leveraging the latent representations of a pretrained autoencoder. Additionally, X-VAE introduces a learnable latent scaling factor that explicitly modulates the sampling variance in the latent space. This approach not only preserves high reconstruction accuracy but also significantly enhances the realism and controllability of generated samples, achieving a flexible trade-off between diversity and fidelity. Empirical evaluations on standard benchmarks demonstrate that X-VAE achieves superior alignment between the learned latent distribution and the empirical data distribution.

This paper addresses the fundamental mismatch between the continuous latent space of standard Variational Autoencoders (VAEs) and the inherently discrete nature of data such as text. To resolve this, we propose the Discrete VAE—a VAE explicitly designed for categorical latent variables. Methodologically, we derive the evidence lower bound (ELBO) rigorously from first principles of variational inference under categorical latents and employ the Gumbel-Softmax reparameterization to enable differentiable gradient estimation in discrete latent spaces. Our key contributions are threefold: (1) a tutorial-style, unified theoretical framework for discrete VAEs; (2) a robust and reproducible training paradigm; and (3) publicly released, fully functional code. Experiments demonstrate that the Discrete VAE significantly improves interpretability and structural coherence in discrete data generation, outperforming continuous-latent baselines while preserving principled probabilistic modeling.

In variational autoencoders (VAEs), the latent dimension must be manually specified, often leading to over-parameterization or under-representation. Method: We propose an adaptive latent dimension selection method based on automatic relevance determination (ARD). By introducing a hierarchical Bayesian prior over the latent space, we jointly model the variances of individual latent axes and optimize the evidence lower bound (ELBO) via variational inference and reparameterization, enabling end-to-end identification of latent dimension importance. Contribution/Results: This work is the first to systematically integrate ARD into the VAE framework without requiring additional hyperparameter tuning, thereby automatically identifying data-driven effective latent dimensions. On multiple benchmark datasets, our method significantly improves FID (average reduction of 12.3%) and disentanglement metrics (DCI disentanglement score increase of 27.6%), effectively mitigating dimensional redundancy while enhancing model interpretability and generalization.

Multimodal variational autoencoders (MVAEs) suffer from overly rigid cross-modal representation coupling, making it difficult to simultaneously ensure high-quality shared representations and modality-specific fidelity. Method: We propose a soft-constrained Mixture-of-Experts (Soft-MoE) prior that replaces hard parameter sharing with learnable gating weights, enabling flexible alignment of modality-specific latent distributions under a unified posterior. This decouples modality-invariant and modality-specific representations while preserving information integrity via variational inference and a soft alignment loss. Contribution/Results: Experiments on multiple benchmarks and real-world multimodal datasets demonstrate that our approach significantly outperforms existing shared-architecture MVAEs. It achieves state-of-the-art performance in both latent representation quality—measured by disentanglement and downstream task accuracy—and missing modality imputation accuracy.

This work addresses the challenge of integrating variational autoencoders (VAEs) as trainable layers within neural networks. It proposes a general framework for flexibly embedding VAEs into arbitrary network architectures, accompanied by an end-to-end training strategy that leverages the reparameterization trick and probabilistic modeling to ensure full differentiability throughout the pipeline. For the first time, this approach enables VAEs to function as plug-and-play modules akin to standard neural network layers, substantially enhancing their compatibility and representational capacity within complex models. Experimental results demonstrate that the proposed VAE layer consistently achieves stable performance across diverse tasks and outperforms conventional standalone VAE models, thereby significantly expanding the applicability of VAEs in deep learning systems.

This work addresses the challenge in variational autoencoders (VAEs) of simultaneously achieving high representational capacity and disentangled, low-dimensional latent representations. The authors formulate VAE training as a soft-constrained optimization problem, introducing an entropy-based soft constraint mechanism to regulate the information content of individual latent variables. Coupled with weight filtering, this approach enables automatic pruning of low-entropy dimensions. The proposed method enhances representation efficiency while preserving disentanglement. Experiments demonstrate significant improvements: on dSprites, activation scores increase by 43–62%, FactorVAE score reaches 0.891, and reconstruction error decreases by 38%; on MNIST, over 90% classification accuracy is achieved using only two latent dimensions—reducing input dimensionality by 80% compared to baselines—and training convergence accelerates by 37%.

Traditional conditional variational autoencoders (CVAEs) suffer from blurry and mode-collapsed image generation, primarily due to the restrictive assumption that the label-conditioned posterior equals a standard normal prior. To address this, we propose a Normalizing Flow-enhanced CVAE framework that replaces the fixed isotropic Gaussian prior with an expressive, invertible flow-based transformation to model complex label-conditioned posteriors. Additionally, we treat the Gaussian decoder’s variance as a learnable parameter to improve reconstruction fidelity and flexibility. Experiments demonstrate that our method achieves a 5% reduction in Fréchet Inception Distance (FID) and a 7.7% improvement in log-likelihood over standard CVAEs, significantly outperforming existing CVAE variants. Crucially, it simultaneously enhances both sample fidelity and diversity—resolving the long-standing trade-off between realism and variability in conditional generative modeling.

Standard variational autoencoders employ Gaussian priors, which struggle to align with data manifolds exhibiting non-Euclidean topologies—such as periodicity or boundedness—leading to distorted representations. This work proposes a topology-aware latent space modeling framework that constructs factorized prior distributions tailored to manifolds decomposable into products of circles, intervals, and lines, along with their finite group quotients. This design enables disentangled latent representations and analytically tractable KL divergences. By integrating differentiable coordinate transformations, group-invariant decoding, and anchor-point constraints, the approach ensures smooth gradients and topological consistency. To our knowledge, this is the first method to systematically align latent variable distributions with the intrinsic topology of data manifolds, supporting reparameterizable encoder–prior pairs and significantly outperforming Gaussian-prior baselines on synthetic manifolds as well as rotation- and cyclic-translation variants of MNIST.

Traditional VAEs restrict the posterior to diagonal covariance structures, limiting their ability to capture correlations among latent variables. To address this, we propose the Full-Covariance Variational Autoencoder (FC-VAE), enabling efficient full-covariance inference while preserving analytical tractability of Gaussian posteriors. Our key innovation is a structured decomposition of the posterior covariance: ( L = C cdot ext{diag}(sigma) ), where ( C ) is a global low-rank coupling matrix and ( ext{diag}(sigma) ) is a sample-specific diagonal scaling matrix. This formulation permits closed-form KL divergence computation and standard reparameterization without auxiliary approximations or sampling. FC-VAE significantly improves reconstruction accuracy (MSE), probabilistic calibration (NLL, Brier score, ECE), and unsupervised clustering performance (NMI, ARI), especially at moderate latent dimensions. Extensive experiments across multiple image datasets validate its effectiveness and generalization capability.