LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation

2026-06-01

AI Summary

This work addresses the accumulation of errors and identity drift in autoregressive long video generation caused by sliding-window attention. To mitigate these issues, the authors propose a retrieval-augmented generation (RAG) framework that treats self-generated latent history as a content-addressable memory bank and incorporates non-local context at each generation step through a lightweight retrieval mechanism. To further enhance temporal consistency, they introduce a windowed temporal difference loss that regularizes the stability of historical representations. This approach achieves dynamic history retrieval within an autoregressive video diffusion model for the first time, consistently improving video quality across various backbone architectures and generation lengths, and attains the best average ranking on the VBench-Long benchmark.

Abstract

Autoregressive (AR) video diffusion enables variable-length synthesis, but long-horizon generation often suffers from accumulated errors and identity drift. For efficiency, existing methods commonly adopt sliding-window attention during generation. This creates an irreversible generation trajectory: once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. We address this limitation by formulating long video generation as a retrieval-augmented generation (RAG) problem. Rather than relying solely on the recent window, we treat previously generated latents as a dynamic, searchable history. We propose LongLive-RAG, a general retrieval framework for AR video generation. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents. This lightweight retrieval step adds only a small overhead relative to generation and lets the generator condition on non-local context instead of only the recent window. To make retrieval more discriminative, we introduce the Window Temporal Delta Loss that suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. Together, these components help reduce error accumulation caused by sliding-window attention. Experiments across multiple AR backbones and generation lengths show improved long-video quality and the best average VBench-Long rank. To our knowledge, among open-ended AR long video generation methods, LongLive-RAG is the first to formulate self-generated latent history as content-addressable retrieval memory. Code is available at https://github.com/qixinhu11/LongLive-RAG.

Problem

• long video generation
• error accumulation
• identity drift
• sliding-window attention
• autoregressive diffusion

Innovation

• Retrieval-Augmented Generation
• Long Video Synthesis
• Autoregressive Diffusion
• Latent Memory Retrieval
• Temporal Delta Loss