Surprise Forcing: What to Remember, When to Skip in Long Video Generation

2026-07-20

AI Summary

This work addresses the challenges of long-range visual information forgetting and uneven computational allocation in long video generation, which stem from fixed denoising schedules and limited context windows. To this end, the authors propose a training-free, online resource allocation framework that dynamically preserves critical content and adaptively allocates denoising steps across video segments. The approach integrates a surprise-gated memory bank and a surprise-aware denoising mechanism, leveraging value token descriptors, global bias, nearest-neighbor novelty signals, and feedback-controlled budgets. Efficient memory management and computation skipping are achieved through priority-based replacement, relevance-aware routing, and a local percentile scheduler. Evaluated on the VBench benchmark, the method significantly improves long-term temporal consistency and visual quality while maintaining real-time streaming throughput.

Abstract

Streaming autoregressive diffusion makes minute-scale video synthesis practical, but its bounded context and fixed denoising schedule allocate resources uniformly across a highly non-stationary sequence. A rolling key-value cache forgets distant visual evidence even when that evidence remains important, while every generated chunk receives the same number of denoising passes irrespective of its actual difficulty. We introduce Surprise Forcing, a training-free framework that treats both limitations as online resource-allocation problems. A Surprise-Gated Memory Bank summarizes evicted frames with value-token descriptors, evaluates them using complementary global-deviation and nearest-neighbor novelty signals, and regulates admission through a feedback-controlled budget in normalized score space. Priority-based replacement and relevance-aware routing then keep the external memory compact and useful. In parallel, Surprise-Aware Denoising estimates chunk difficulty from the maximum adjacent-frame cosine distance after the first denoising pass and uses a local percentile scheduler to skip intermediate steps for comparatively easy chunks. Experiments on VBench, VBench-Long, and VBench-2.0 show that the proposed allocation strategy improves long-horizon consistency and visual quality while retaining real-time streaming throughput.

Problem

• long video generation
• resource allocation
• memory forgetting
• denoising schedule
• streaming diffusion

Innovation

• Surprise Forcing
• Memory Bank
• Streaming Video Generation
• Adaptive Denoising
• Resource Allocation