vae layer integration

This work addresses the challenge of integrating variational autoencoders (VAEs) as trainable layers within neural networks. It proposes a general framework for flexibly embedding VAEs into arbitrary network architectures, accompanied by an end-to-end training strategy that leverages the reparameterization trick and probabilistic modeling to ensure full differentiability throughout the pipeline. For the first time, this approach enables VAEs to function as plug-and-play modules akin to standard neural network layers, substantially enhancing their compatibility and representational capacity within complex models. Experimental results demonstrate that the proposed VAE layer consistently achieves stable performance across diverse tasks and outperforms conventional standalone VAE models, thereby significantly expanding the applicability of VAEs in deep learning systems.

This paper addresses the conceptual gap between variational autoencoders (VAEs) and vector-quantized VAEs (VQ-VAEs) in modeling continuous versus discrete latent representations. Methodologically, it proposes a novel autoencoder framework that eliminates both the KL divergence term and the reparameterization trick; instead, it explicitly enforces latent space compactness via learnable clustering centers and employs multi-vector outputs to jointly support continuous interpolation and discrete reconstruction. Key contributions include: (1) uncovering an intrinsic relationship between autoencoder generative fidelity and latent space compactness; (2) establishing a deterministic transition from VAEs to VQ-VAEs without relying on probabilistic assumptions; and (3) empirically validating smooth interpolation and stable reconstruction on MNIST, CelebA, and FashionMNIST. Experiments further reveal that naively increasing the number of output vectors leads to model degradation—manifesting as localized, patchwise discrete encoding—highlighting the critical role of architectural design.

This paper addresses the fundamental mismatch between the continuous latent space of standard Variational Autoencoders (VAEs) and the inherently discrete nature of data such as text. To resolve this, we propose the Discrete VAE—a VAE explicitly designed for categorical latent variables. Methodologically, we derive the evidence lower bound (ELBO) rigorously from first principles of variational inference under categorical latents and employ the Gumbel-Softmax reparameterization to enable differentiable gradient estimation in discrete latent spaces. Our key contributions are threefold: (1) a tutorial-style, unified theoretical framework for discrete VAEs; (2) a robust and reproducible training paradigm; and (3) publicly released, fully functional code. Experiments demonstrate that the Discrete VAE significantly improves interpretability and structural coherence in discrete data generation, outperforming continuous-latent baselines while preserving principled probabilistic modeling.

Traditional VAEs restrict the posterior to diagonal covariance structures, limiting their ability to capture correlations among latent variables. To address this, we propose the Full-Covariance Variational Autoencoder (FC-VAE), enabling efficient full-covariance inference while preserving analytical tractability of Gaussian posteriors. Our key innovation is a structured decomposition of the posterior covariance: ( L = C cdot ext{diag}(sigma) ), where ( C ) is a global low-rank coupling matrix and ( ext{diag}(sigma) ) is a sample-specific diagonal scaling matrix. This formulation permits closed-form KL divergence computation and standard reparameterization without auxiliary approximations or sampling. FC-VAE significantly improves reconstruction accuracy (MSE), probabilistic calibration (NLL, Brier score, ECE), and unsupervised clustering performance (NMI, ARI), especially at moderate latent dimensions. Extensive experiments across multiple image datasets validate its effectiveness and generalization capability.

Variational autoencoders (VAEs) lack non-asymptotic convergence guarantees, limiting theoretical understanding of their optimization dynamics. Method: Leveraging stochastic optimization and variational inference theory, this work establishes the first unified non-asymptotic convergence analysis framework for VAE training under SGD and Adam. Contribution/Results: The analysis yields a convergence rate of (O(log n / sqrt{n})) for canonical VAE variants—including linear VAEs, deep Gaussian VAEs, (eta)-VAEs, and importance-weighted autoencoders (IWAEs)—under standard objective functions. Crucially, the bound explicitly quantifies how key hyperparameters—such as batch size and number of variational samples—affect convergence, transcending prior asymptotic analyses. This work fills a fundamental gap in VAE theory and provides rigorous mathematical foundations for interpretability, stability, and principled hyperparameter design.

Traditional variational autoencoders (VAEs) are constrained by a standard isotropic Gaussian prior, which often fails to accurately capture the complex latent structure of real-world data, leading to suboptimal generation quality and reconstruction fidelity. To address this limitation, this work proposes the X-VAE framework, which innovatively constructs a data-adaptive Gaussian mixture prior by leveraging the latent representations of a pretrained autoencoder. Additionally, X-VAE introduces a learnable latent scaling factor that explicitly modulates the sampling variance in the latent space. This approach not only preserves high reconstruction accuracy but also significantly enhances the realism and controllability of generated samples, achieving a flexible trade-off between diversity and fidelity. Empirical evaluations on standard benchmarks demonstrate that X-VAE achieves superior alignment between the learned latent distribution and the empirical data distribution.

This work addresses the amortization gap in variational autoencoders (VAEs), which arises from sharing encoder parameters across all inputs and leads to inaccurate posterior approximations. To mitigate this limitation, the authors propose the Instance-Adaptive Variational Autoencoder (IA-VAE), which employs a hypernetwork to dynamically generate input-specific modulation parameters for the encoder. This enables instance-dependent adaptation of the inference model within a single forward pass, enhancing model expressiveness and parameter efficiency without sacrificing computational tractability. Empirical evaluations on both synthetic data and standard image benchmarks demonstrate that IA-VAE achieves more accurate posterior approximations and higher test-set evidence lower bounds (ELBOs), effectively narrowing the amortization gap.

This work addresses the slow convergence of diffusion model training, a challenge exacerbated by existing acceleration methods that rely on external encoders or dual-model architectures, incurring substantial computational overhead. To overcome this, the authors propose a lightweight, endogenous guidance framework that leverages the intrinsic visual priors of a pretrained VAE. By introducing a lightweight projection layer, the method aligns the VAE’s reconstruction features with intermediate representations in a diffusion Transformer, guided by a dedicated feature alignment loss. Notably, this approach requires no additional models and incurs only a 4% increase in GFLOPs. Experiments across multiple benchmarks demonstrate significant improvements in both training convergence speed and generation quality, matching or surpassing state-of-the-art acceleration techniques while maintaining simplicity, generality, and computational efficiency.

This work addresses the training instability and codebook collapse in vector-quantized variational autoencoders (VQ-VAEs), which arise from the tight coupling between representation learning and codebook optimization. To resolve this, the authors propose the VP-VAE framework, which decouples the quantization operation by modeling it as an adaptive perturbation in the latent space, thereby eliminating the need for an explicit codebook. Leveraging Metropolis–Hastings sampling, the method generates distribution-consistent and scale-adaptive perturbations. Under the assumption of uniformly distributed latent variables, a lightweight variant termed FSP is derived, offering both a unified theoretical interpretation and practical enhancements for fixed quantizers. Experiments demonstrate that the proposed approach significantly improves reconstruction fidelity on image and audio tasks, promotes more balanced token usage, and enhances training stability and robustness.

This work addresses the limitation of conventional variational autoencoders (VAEs), whose encoders—constrained by the reparameterization trick—struggle to model complex posterior distributions. To overcome this, the authors propose a novel encoder that, for the first time, integrates a diffusion model into the VAE encoding process. They further introduce an alternating training strategy inspired by the Expectation-Maximization (EM) algorithm, which effectively aligns the optimization objectives of the encoder and decoder, thereby ensuring reliable synchronization in the latent space. The proposed approach preserves the simplicity and efficiency of standard diffusion model training while substantially enhancing the model’s capacity to capture complex data distributions and improving reconstruction quality.