KVAE: Family of Tokenizers for Multimodal Generative Models

2026-08-06

AI Summary

Existing multimodal generative models are hindered by the absence of an efficient, unified, and high-quality tokenizer, which limits both generation speed and sample fidelity. This work proposes KVAE—a family of multimodal tokenizers based on variational autoencoders—that enables high-fidelity compression of full-band audio, images, and causal video within a unified architecture. KVAE employs continuous latent representations and tailors compression ratios and channel dimensions per modality to suit diffusion model training. Experimental results demonstrate that KVAE matches or surpasses current open-source state-of-the-art methods across reconstruction metrics—including PSNR, LPIPS, and PESQ—as well as generation quality benchmarks such as FID and CLIP/CLAP scores, corroborated by subjective evaluations. The study also provides complete training protocols and comprehensive ablation studies.

Abstract

Latent diffusion modeling (LDM), a prominent paradigm, utilizes tokenizers to map input signal to compressed representation. This dependency positions tokenizer as an integral part of generation process itself, since it affects learning speed, quality of synthesized samples and lay foundation for later applications. This report presents series of KVAE tokenizers for audio, image and video, all designed for subsequent text-conditioned generation: KVAE-Audio, a continuous full-band 48 kHz tokenizer with a 50 Hz latent of 64 channels; KVAE-3D -- two causal video tokenizers for 4x16x16 and 4x8x8 compression; KVAE-2D, an image model, compressing input by factor of 8 with 32 channels. We demonstrate that reconstruction (PSNR, LPIPS, PESQ, etc.) and generation results on objective (Frechet Distance, CLIP score, CLAP score, etc.) and subjective (side-by-side evaluation) metrics matches or surpasses frontier opensource tokenizers, such as VAEs from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio. Considering difficulty of development, we share with community training details, model selection method and ablation on design choices. The code is publicly available at https://github.com/kandinskylab/kvae and https://github.com/kandinskylab/kvae-audio.

Problem

• tokenizer
• multimodal generative models
• latent representation
• text-conditioned generation
• compression

Innovation

• KVAE
• multimodal tokenization
• latent diffusion models
• causal video tokenizer
• text-conditioned generation