How Neural Losses Shape VAE Latents

2026-05-30

AI Summary

This study investigates how neural reconstruction objectives—such as perceptual and adversarial losses—reshape the informational content and geometric structure of the latent space in variational autoencoders (VAEs). Building upon the β-VAE framework and combining theoretical analysis with empirical validation, this work provides the first mechanistic account demonstrating that neural losses systematically reduce the information content of latent variables, enhance isotropy in the latent space, and promote more uniform posterior variances across dimensions. These findings challenge the explanatory power of classical rate–distortion theory in describing VAE behavior and elucidate how neural reconstruction losses fundamentally reconfigure the rate–distortion trade-off. The results offer a novel perspective for understanding the latent space properties of deep generative models.

Abstract

Modern VAEs are rarely trained with the pointwise likelihood implied by the standard $β$-VAE objective. In practice, pointwise reconstruction is often combined with perceptual and adversarial losses, despite a lack of understanding of how this changes the latent dynamics of the model. We show that the choice of reconstruction loss reshapes the rate-distortion problem itself, altering both the information content and the geometry of the learned latent space in ways that may be invisible from reconstructions alone. First, we prove and verify empirically that augmenting pointwise reconstruction with neural terms, such as perceptual and adversarial objectives, reduces the amount of information stored in the latent representations. Second, we show that neural reconstruction losses systematically change the geometry of the latent space: they make representations more isotropic and distribute uncertainty more evenly across latent dimensions, producing different posterior variance profiles. These findings highlight how the rate-distortion tradeoff is not a comprehensive lens to understand the behavior of VAEs, and we propose a more mechanistic approach to investigate how the choice of a distortion metric reshapes the optimization problem.

Problem

• VAE
• neural losses
• latent space
• rate-distortion
• reconstruction loss

Innovation

• neural losses
• latent geometry
• rate-distortion tradeoff
• perceptual loss
• variational autoencoders