MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation

2026-06-08

AI Summary

This work addresses the computational burden and long-range consistency challenges in long video generation caused by excessively long sequences. The authors propose a coarse-to-fine generation framework based on hierarchical latent representations. Their approach first compresses video frames into multi-scale tokens using a hierarchical autoencoder, then employs a video diffusion model to generate content level-by-level—modeling global structure first and progressively refining local details. This strategy substantially reduces the computational cost of modeling high-frequency details over long ranges while effectively preserving key semantic consistencies such as scene layout and object persistence. Evaluated on a newly curated long-duration Minecraft video dataset, the method significantly outperforms existing baselines, demonstrating superior long-range stability in both geometric structure and object consistency.

Abstract

Video generative models have become increasingly powerful, but long-range consistency remains challenging to achieve because even a few dozen frames require impractically long transformer sequence lengths. We show that this issue can be mitigated by generating video using coarse-to-fine rollout within a multi-scale token space. Our approach is simple: first, we pre-train an autoencoder that compresses each frame into a hierarchy of tokens, with levels ranging from the typical latent resolution to only a handful of tokens per frame. The coarsest levels capture the most consequential information, such as scene layout and semantics, while finer levels add high-frequency appearance and texture. Then, we train a video diffusion model to generate these tokens using coarse-to-fine rollout. By carefully controlling the level of detail at which frames are generated and used as context during each rollout step, we are able to preserve long-range consistency in geometry and object permanence while spending less compute on the long-range consistency of less perceptually relevant details. We validate this approach using a custom dataset of long Minecraft videos, where it produces substantially more consistent rollouts compared to existing baselines.

Problem

• long-range consistency
• video generation
• transformer sequence length
• temporal coherence
• video diffusion

Innovation

• hierarchical latents
• coarse-to-fine rollout
• long-range consistency
• video diffusion model
• multi-scale token space