MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation

2026-10-01

AI Summary

This study addresses the limitation of long-horizon autoregressive video generation imposed by finite context windows, which leads to irreversible loss of object details. To overcome this, we propose a spatiotemporal memory mechanism that dynamically assembles historical key-value (KV) pairs into a mosaic memory via a lightweight router. Notably, the generator remains frozen while only the routing module is trained to select non-contiguous historical KV pairs through sparse attention, enabling visual content recovery within a fixed computational budget. Furthermore, we introduce RememBench, a dedicated evaluation benchmark for this task. Experimental results demonstrate that our approach significantly outperforms sliding-window and block-retrieval baselines in both text-to-video and image-to-video generation, substantially improving long-range revisit consistency.

Abstract

Long-horizon autoregressive video generation is limited by a finite context window. When an object or scene falls out of context, its fine-grained visual details may be lost and difficult to recover upon reappearance. To retain access to such visual details, we introduce MosaiChunk, a spatio-temporal memory mechanism that composes a mosaic of selected historical key-value (KV) entries across space and time. Our approach is motivated by the observation that a frozen video generator can directly consume such non-contiguous historical KV and recover the corresponding visual content. We therefore keep the generator fixed and learn only a lightweight router that determines which historical sections to include in the mosaic under a fixed active-memory budget. We further introduce RememBench, a benchmark of long-horizon revisits with prompt-driven text-to-video (T2V) and camera-driven image-to-video (I2V) splits. Our experiments show that MosaiChunk consistently improves revisit consistency over both sliding-window inference and whole-chunk retrieval under matched memory budgets, across both T2V and I2V settings.

Problem

• autoregressive video generation
• long-horizon
• context window limitation
• visual detail recovery
• revisit consistency

Innovation

• Spatio-Temporal Memory
• Autoregressive Video Generation
• Key-Value Routing
• Lightweight Router
• Long-horizon Consistency