kl annealing

This work addresses the challenge in variational autoencoders (VAEs) of simultaneously achieving high representational capacity and disentangled, low-dimensional latent representations. The authors formulate VAE training as a soft-constrained optimization problem, introducing an entropy-based soft constraint mechanism to regulate the information content of individual latent variables. Coupled with weight filtering, this approach enables automatic pruning of low-entropy dimensions. The proposed method enhances representation efficiency while preserving disentanglement. Experiments demonstrate significant improvements: on dSprites, activation scores increase by 43–62%, FactorVAE score reaches 0.891, and reconstruction error decreases by 38%; on MNIST, over 90% classification accuracy is achieved using only two latent dimensions—reducing input dimensionality by 80% compared to baselines—and training convergence accelerates by 37%.

This study addresses the closed-form expression of the Kullback–Leibler (KL) divergence between Gaussian priors and posteriors in variational autoencoders (VAEs) and its role in training dynamics. Drawing on information theory and probability, the work systematically derives analytical solutions for the KL divergence under both univariate and diagonal-covariance multivariate Gaussian assumptions. It further elucidates how individual terms in the KL divergence contribute to regularization in the latent space and shape the model’s generative capacity. By providing a clear and rigorous theoretical derivation, this research deepens the understanding of the intrinsic nature of the VAE’s regularization term and offers principled guidance for model design and practical optimization.

Variational autoencoders (VAEs) often suffer from posterior collapse, degrading generative diversity; existing mitigation strategies rely on regularization trade-offs or architectural constraints, limiting generalizability and controllability. This paper proposes an architecture-agnostic method for localized posterior collapse control: we define a local collapse metric and introduce a latent reconstruction loss (LRL), leveraging the mathematical properties of injection and composition functions to enable end-to-end optimization within the variational inference framework. LRL requires no architectural modifications and jointly preserves reconstruction fidelity and latent identifiability. Experiments on MNIST, FashionMNIST, Omniglot, CelebA, and FFHQ demonstrate that our approach significantly alleviates posterior collapse, markedly improving sample diversity and distribution coverage. The method establishes a more robust and generalizable control paradigm for VAE training.

This work addresses posterior collapse in variational autoencoders (VAEs), a fundamental failure mode in representation learning. Leveraging tools from high-dimensional statistical physics and variational inference theory, we construct a minimal VAE model and rigorously analyze its large-sample asymptotics. We establish, for the first time, the existence of a data-size-independent critical β-value: when the KL-weight β exceeds this threshold, posterior collapse to the prior becomes inevitable. Concurrently, the rate-distortion curve exhibits explicit sample-size dependence, with asymptotic performance improving markedly as data volume grows. Our analysis reveals that the phase transition threshold is jointly governed by β and dataset size, precisely characterizing both collapse conditions and the rate-distortion trade-off. These theoretical predictions are robustly validated across diverse nonlinear VAE architectures and real-world datasets, challenging conventional qualitative interpretations of β-regularization.

Variational autoencoders (VAEs) lack non-asymptotic convergence guarantees, limiting theoretical understanding of their optimization dynamics. Method: Leveraging stochastic optimization and variational inference theory, this work establishes the first unified non-asymptotic convergence analysis framework for VAE training under SGD and Adam. Contribution/Results: The analysis yields a convergence rate of (O(log n / sqrt{n})) for canonical VAE variants—including linear VAEs, deep Gaussian VAEs, (eta)-VAEs, and importance-weighted autoencoders (IWAEs)—under standard objective functions. Crucially, the bound explicitly quantifies how key hyperparameters—such as batch size and number of variational samples—affect convergence, transcending prior asymptotic analyses. This work fills a fundamental gap in VAE theory and provides rigorous mathematical foundations for interpretability, stability, and principled hyperparameter design.

Posterior collapse in variational autoencoders (VAEs) is commonly attributed to optimization failure, obscuring its fundamental origins. Method: We formulate posterior collapse as a phase transition phenomenon jointly driven by data structure and model hyperparameters, drawing on statistical physics principles; we analyze the stability of the KL divergence between the approximate posterior and prior via variational inference, and identify a critical point where this divergence exhibits discontinuity—marking the phase boundary between collapse and effective latent inference. Contribution/Results: Our framework systematically characterizes the phase transition behavior on both synthetic and real-world datasets, precisely identifying hyperparameter thresholds that prevent collapse. This work establishes a novel theoretical foundation for understanding trainability, representational capacity, and latent-space geometry in deep generative models, offering actionable guidance for VAE design and hyperparameter selection.

This work addresses the challenge of integrating variational autoencoders (VAEs) as trainable layers within neural networks. It proposes a general framework for flexibly embedding VAEs into arbitrary network architectures, accompanied by an end-to-end training strategy that leverages the reparameterization trick and probabilistic modeling to ensure full differentiability throughout the pipeline. For the first time, this approach enables VAEs to function as plug-and-play modules akin to standard neural network layers, substantially enhancing their compatibility and representational capacity within complex models. Experimental results demonstrate that the proposed VAE layer consistently achieves stable performance across diverse tasks and outperforms conventional standalone VAE models, thereby significantly expanding the applicability of VAEs in deep learning systems.

This work addresses the pervasive issue of posterior collapse in variational autoencoders (VAEs), which renders latent codes uninformative due to two intertwined mechanisms: gradient imbalance and the information gap. To mitigate both simultaneously, the authors propose λ-VAE, a novel approach that applies dimension-wise exponential scaling to the noise in the reparameterization trick while preserving the original KL divergence penalty. This design achieves variance balancing across latent dimensions, effectively suppressing collapse. Notably, λ-VAE provides the first unified framework that concurrently alleviates both collapse mechanisms and admits a closed-form optimal solution for the scaling exponents. Empirical evaluations demonstrate that λ-VAE substantially reduces the number of collapsed dimensions across multiple benchmark datasets, yielding up to a 2.8-nat increase in information capacity and up to a 0.33 bits-per-dim improvement in reconstruction quality.

This work addresses the posterior collapse problem in variational autoencoders (VAEs), wherein latent variables become ineffective due to degenerate posteriors. The authors propose a novel historical consensus training strategy that iteratively selects Gaussian mixture model (GMM) priors and constructs stable parameter regions through alternating optimization, thereby fundamentally preventing collapse. For the first time, this approach leverages the diversity of GMM clustering to form a historical barrier that excludes collapsed solutions without requiring architectural constraints or explicit stability conditions. Experimental results demonstrate that the method consistently learns non-collapsed representations across both synthetic and real-world datasets, exhibiting strong robustness to decoder variance and regularization strength, and is readily applicable to arbitrary neural network architectures.

This work addresses the training instability and codebook collapse in vector-quantized variational autoencoders (VQ-VAEs), which arise from the tight coupling between representation learning and codebook optimization. To resolve this, the authors propose the VP-VAE framework, which decouples the quantization operation by modeling it as an adaptive perturbation in the latent space, thereby eliminating the need for an explicit codebook. Leveraging Metropolis–Hastings sampling, the method generates distribution-consistent and scale-adaptive perturbations. Under the assumption of uniformly distributed latent variables, a lightweight variant termed FSP is derived, offering both a unified theoretical interpretation and practical enhancements for fixed quantizers. Experiments demonstrate that the proposed approach significantly improves reconstruction fidelity on image and audio tasks, promotes more balanced token usage, and enhances training stability and robustness.