Self Gradient Forcing: Native Long Video Extrapolation

2026-07-22

AI Summary

This work addresses the insufficient long-term extrapolation performance—manifested as degraded identity preservation, spatial layout coherence, and temporal consistency—in autoregressive video diffusion models, which stems from a lack of gradient supervision between historical context and future frames (termed the “history-context gradient gap”). To bridge this gap, the authors propose a two-stage training strategy: in the first stage, gradient-free autoregressive forward passes generate and cache intermediate states; in the second stage, contextual key-value representations are reconstructed in parallel from these cached states, and gradients from future-frame reconstruction losses are backpropagated to supervise the encoding of historical context. This approach uniquely restores supervisory signals for critical context writing without requiring full-sequence backpropagation, thereby refining the causal memory mechanism. Experiments demonstrate that, trained on only 5-second clips, the method extrapolates videos lasting several minutes while significantly outperforming existing approaches in identity fidelity, background consistency, and temporal stability.

Abstract

Recent autoregressive video diffusion methods are increasingly built upon Self Forcing, where the student is trained on histories produced by its own rollout rather than ground-truth video contexts. This reduces exposure bias, but the historical key-value cache is still used by future frames only as frozen rollout state. As a result, future losses cannot supervise how earlier generated latents should be written into more useful keys and values for later video-latent generation. We call this the historical context-gradient gap. We propose Self Gradient Forcing (SGF), a two-pass training strategy that restores this missing supervision signal without backpropagating through the full serial rollout. Pass 1 performs a no-gradient autoregressive rollout matching inference and, at a sampled denoising exit step, records both the self-generated context and the noisy latents fed to the model. Pass 2 performs parallel context-gradient reconstruction for the recorded exit step. The generated context is used as stop-gradient clean-latent input, while the model recomputes the context KV representations and future-to-context causal attention. Thus, SGF provides the missing memory-writing supervision within the native autoregressive training objective, using losses on future video latents to train the model to encode context into more effective causal memory. Across extensive long-horizon frame-wise and chunk-wise experiments under different initializations, SGF achieves stronger native long-video extrapolation than Self Forcing, especially in subject identity, background/layout consistency, and temporal stability. Remarkably, using only a 5-second training window, SGF can extrapolate to videos lasting several minutes. Code and models will be released to advance research on autoregressive video generation.

Problem

• exposure bias
• historical context-gradient gap
• autoregressive video generation
• long video extrapolation
• causal memory

Innovation

• Self Gradient Forcing
• autoregressive video diffusion
• long-video extrapolation
• context-gradient supervision
• causal memory