OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

2026-07-26

AI Summary

This work addresses the challenge of fine-grained cross-modal alignment in audio-visual joint generation, which arises from structural discrepancies between modalities. To tackle this issue, the authors propose OmniVAE, a novel framework that achieves fine-grained semantic alignment in the latent space of audio and video within a variational autoencoder (VAE). OmniVAE jointly trains modality-specific VAEs while incorporating clip-level contrastive learning and knowledge distillation from pretrained modality-specific semantic encoders to enhance cross-modal consistency. Experimental results demonstrate that OmniVAE significantly improves both the quality of text-to-audio-visual generation and the accuracy of cross-modal synchronization in downstream tasks, thereby validating the critical role of unified semantic representations in holistic multimodal generative modeling.

Abstract

Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1

Problem

• audio-video generation
• cross-modal alignment
• latent space
• multimodal synchronization
• joint generation

Innovation

• cross-modal alignment
• joint audio-video generation
• contrastive learning
• latent space distillation
• omnimodal VAE