REDUCIO! Generating 1024$ imes$1024 Video within 16 Seconds using Extremely Compressed Motion Latents

2024-11-20

AI Summary

Commercial video generation models produce high-fidelity results but remain inaccessible due to prohibitive training and inference costs, especially for high-resolution video synthesis. Method: We propose an image-conditioned VAE that compresses video into an ultra-compact motion latent space—achieving 64× latent compression—and integrate it with a two-stage diffusion architecture (text → image → video) to efficiently generate 1024×1024 videos. Our approach introduces a novel motion-content disentangled latent representation and models temporal redundancy as sparse dynamic changes. Contribution/Results: This is the first method to generate 1024×1024 videos in just 15.5 seconds on a single A100 GPU. Training completes in only 3,200 GPU-hours. The framework achieves state-of-the-art visual quality while drastically reducing computational overhead, significantly enhancing efficiency and scalability for high-resolution video generation.

Technology Category

Computer Vision: Diffusion Models for VisionMachine Learning: Deep Generative Models & AutoencodersNatural Language Processing: Generation

Application Category

Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applicationsWeb Mining and Content Analysis: Large pretrained models with web dataSearch and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Commercial video generation models have exhibited realistic, high-fidelity results but are still restricted to limited access. One crucial obstacle for large-scale applications is the expensive training and inference cost. In this paper, we argue that videos contain much more redundant information than images, thus can be encoded by very few motion latents based on a content image. Towards this goal, we design an image-conditioned VAE to encode a video to an extremely compressed motion latent space. This magic Reducio charm enables 64x reduction of latents compared to a common 2D VAE, without sacrificing the quality. Training diffusion models on such a compact representation easily allows for generating 1K resolution videos. We then adopt a two-stage video generation paradigm, which performs text-to-image and text-image-to-video sequentially. Extensive experiments show that our Reducio-DiT achieves strong performance in evaluation, though trained with limited GPU resources. More importantly, our method significantly boost the efficiency of video LDMs both in training and inference. We train Reducio-DiT in around 3.2K training hours in total and generate a 16-frame 1024*1024 video clip within 15.5 seconds on a single A100 GPU. Code released at https://github.com/microsoft/Reducio-VAE .

Problem

• Reducing video generation costs via compressed motion latents
• Enhancing efficiency in high-resolution video synthesis
• Minimizing training and inference time for video LDMs

Innovation

• Extremely compressed motion latents encoding
• Image-conditioned VAE for video compression
• Two-stage text-image-to-video generation