vq-vae training

Existing visual world models suffer from geometric and semantic degradation due to Gaussian bottlenecks, which hinder the preservation of 3D structure and physically consistent camera dynamics in compressed representations. This work proposes the S²VAE framework, which introduces, for the first time, a product-form Power Spherical distribution into the bottleneck of a variational autoencoder, replacing conventional Gaussian latent variables with hyperspherical ones to explicitly align the latent space topology with 3D geometric structure. By adhering to a geometry-first principle, the model learns scene depth, camera pose, and point-level structure directly from observations. Evaluated under high compression rates, S²VAE significantly outperforms traditional approaches, demonstrating the efficacy of geometrically aligned latent spaces in tasks including depth estimation, camera pose recovery, and point cloud reconstruction.

To address the inefficiency in modeling high-dimensional latent spaces and the substantial computational overhead during training and inference in video generation, this paper proposes the Four-Plane Variational Autoencoder (4P-VAE). The method introduces a novel four-plane factorized latent space architecture, projecting spatiotemporal video volumes onto four orthogonal 2D planes. This design enables sublinear growth of latent dimensionality with respect to input resolution while preserving representation fidelity under high compression ratios. 4P-VAE natively supports diverse downstream tasks—including class-conditional generation, frame prediction, and video interpolation—and integrates seamlessly with latent diffusion models (LDMs) for joint training. Experiments demonstrate that 4P-VAE achieves high-fidelity video reconstruction while significantly accelerating LDM training and inference and reducing GPU memory consumption. Overall, it establishes a new paradigm for efficient latent-space modeling of high-dimensional temporal data.

To address the inefficiency in spatiotemporal modeling, high computational complexity, and difficulty in jointly capturing video-level and pixel-level semantics in video representation learning, this paper proposes the Recurrent Vision Model (RVM)—a lightweight video self-encoding framework built upon a recurrent Transformer. Its core innovation is the first-ever recurrent video masked autoencoding paradigm, which performs temporal aggregation of dense frame-wise features and asymmetric masked reconstruction to enable joint spatiotemporal modeling. RVM achieves linear-complexity feature propagation over long sequences, improves parameter efficiency by 30× over SOTA methods, and supports end-to-end training without knowledge distillation. On action recognition and object tracking, RVM matches VideoMAE and V-JEPA; on geometric and dense spatial understanding tasks, it surpasses DINOv2—particularly with compact architectures. Visualization confirms its capability for unified semantic, structural, and motion representation.

This work addresses two key bottlenecks in self-supervised representation learning for point cloud videos: poor generalization of motion modeling and difficulty in disentangling 4D spatiotemporal features. To this end, we propose SD-MAE—the first self-disentangling masked autoencoder framework for point cloud video understanding. Methodologically, SD-MAE introduces implicit motion representation learning in latent space, circumventing biases inherent in explicit motion estimation; it further designs a synergistic mechanism between implicit tokens and geometric tokens to enable multi-granularity disentanglement and reconstruction of hierarchical 4D features. Compared to conventional MAE-based approaches, SD-MAE significantly bridges the representational gap for 4D data, enhancing transferability of pre-trained representations. Extensive experiments demonstrate state-of-the-art performance across five benchmarks—including HOI4D—outperforming both existing self-supervised and fully supervised methods. Notably, SD-MAE achieves a 3.8% absolute improvement in action segmentation accuracy on HOI4D.

This paper addresses the conceptual gap between variational autoencoders (VAEs) and vector-quantized VAEs (VQ-VAEs) in modeling continuous versus discrete latent representations. Methodologically, it proposes a novel autoencoder framework that eliminates both the KL divergence term and the reparameterization trick; instead, it explicitly enforces latent space compactness via learnable clustering centers and employs multi-vector outputs to jointly support continuous interpolation and discrete reconstruction. Key contributions include: (1) uncovering an intrinsic relationship between autoencoder generative fidelity and latent space compactness; (2) establishing a deterministic transition from VAEs to VQ-VAEs without relying on probabilistic assumptions; and (3) empirically validating smooth interpolation and stable reconstruction on MNIST, CelebA, and FashionMNIST. Experiments further reveal that naively increasing the number of output vectors leads to model degradation—manifesting as localized, patchwise discrete encoding—highlighting the critical role of architectural design.

This work addresses the computational inefficiency of conventional video synthesis methods in long-horizon motion generation by proposing an efficient generative framework based on highly compressed motion embeddings. The approach learns an implicit motion representation from large-scale trajectory data, achieving a temporal compression ratio of 64×, and constructs a conditional flow-matching model within this compressed latent space to flexibly respond to text prompts or spatial perturbations. Notably, it is the first method to directly model dynamics in the compressed embedding space, substantially improving both generation efficiency and controllability. Experiments demonstrate that the generated motions surpass state-of-the-art video generation models and specialized motion synthesis approaches in terms of realism, diversity, and computational efficiency.

Current video models exhibit limitations in temporal understanding and heavily rely on large-scale datasets with language supervision, resulting in high training costs and constrained concept learning. This work proposes motion as a core modality, introducing point trajectories—structured motion cues—as an independent input for the first time. By employing a masked autoencoder to reconstruct occluded trajectories, the method enables self-supervised video representation learning without requiring language annotations or extensive appearance-based data. This approach substantially enhances temporal perception and data efficiency. The learned TIME embeddings achieve state-of-the-art performance in zero-shot settings, using four orders of magnitude less training data than existing methods.

This work addresses key limitations of vector-quantized variational autoencoders—namely, the non-differentiability of quantizers, reliance on straight-through estimators, and codebook collapse—by introducing PCA-VAE. The proposed method replaces conventional vector quantization with a differentiable PCA bottleneck trained online via Oja’s rule, thereby eliminating the need for a discrete codebook, commitment loss, or lookup noise. This approach learns orthogonal latent representations ordered by variance and, to our knowledge, is the first to integrate differentiable PCA into generative modeling as a quantization alternative. On CelebA-HQ, PCA-VAE achieves superior reconstruction quality compared to VQ-GAN and SimVQ while using only 1/10 to 1/100 of the latent bit budget, and naturally yields semantically interpretable latent dimensions corresponding to attributes such as pose, lighting, and gender.

This work addresses the instability and degraded generation performance in video variational autoencoders when used with latent diffusion models, which arises from an excessive number of latent channels. To mitigate this issue, the authors propose a frequency-aware latent space compression method that selectively attenuates high-frequency components in the video latent representations, replacing conventional channel pruning. This approach preserves essential structural information while achieving the same compression ratio. The proposed method substantially improves reconstruction fidelity, enhances training stability of the diffusion model, and outperforms strong baseline methods in generation quality, thereby demonstrating the efficacy and advantages of frequency-guided compression for video generation tasks.

Standard variational autoencoders employ Gaussian priors, which struggle to align with data manifolds exhibiting non-Euclidean topologies—such as periodicity or boundedness—leading to distorted representations. This work proposes a topology-aware latent space modeling framework that constructs factorized prior distributions tailored to manifolds decomposable into products of circles, intervals, and lines, along with their finite group quotients. This design enables disentangled latent representations and analytically tractable KL divergences. By integrating differentiable coordinate transformations, group-invariant decoding, and anchor-point constraints, the approach ensures smooth gradients and topological consistency. To our knowledge, this is the first method to systematically align latent variable distributions with the intrinsic topology of data manifolds, supporting reparameterizable encoder–prior pairs and significantly outperforming Gaussian-prior baselines on synthetic manifolds as well as rotation- and cyclic-translation variants of MNIST.