Mitigating Compounding Error via Video Representation Regularization

2026-07-29

AI Summary

This work addresses the degradation in long-term video generation quality within autoregressive frameworks, which stems from error accumulation due to dimensional collapse in hidden representations under sliding window mechanisms. The study establishes, for the first time, a strong correlation between generation drift and the decline in the effective rank (erank) of latent representations. To mitigate this, the authors propose a lightweight representation regularization strategy that stabilizes the latent space and suppresses iterative error propagation. Notably, the research also demonstrates that conventional data augmentation fails to alleviate this issue, challenging prevailing assumptions. Evaluated within a video diffusion world model, the proposed method substantially improves both Aesthetic Quality (from 38.65 to 55.56) and Imaging Quality (from 44.37 to 72.08) on VBench, significantly enhancing the stability of long-sequence generation.

Abstract

Video diffusion-based world models enable long autoregressive video generation for robotics, autonomous driving and simulation tasks, yet sliding-window autoregressive inference suffers from severe error accumulation that degrades frame quality over time. Although this phenomenon has been widely observed, the underlying mechanism of compounding error and how to achieve stable long-horizon generation remain largely unresolved. In this paper, we investigate the internal representation dynamics of video world models and discover that compounding error is tightly coupled with dimensional collapse of hidden representations. Specifically, the effective rank of model representations sharply decreases at the onset of generation drift, revealing a strong connection between representational degradation and long-term rollout instability. Furthermore, we find that pure training data scaling fails to boost model resistance to error drift, contradicting mainstream scaling paradigms. To address this problem, we propose video representation regularization, a lightweight training constraint that stabilizes latent representations and suppresses iterative error accumulation. Compared with Diffusion Forcing, our method achieves improvements from 38.65 to 55.56 and from 44.37 to 72.08 on the Aesthetic Quality and Imaging Quality metrics of VBench. Our work establishes the first connection between autoregressive video drifting and model internal representations, adopts erank as a quantitative metric for error accumulation, reveals counterintuitive scaling limitations for video world models, and presents a simple yet effective regularization strategy to improve long video generation robustness.

Problem

• compounding error
• video diffusion models
• autoregressive generation
• representation collapse
• long-horizon video generation

Innovation

• compounding error
• dimensional collapse
• video representation regularization
• effective rank
• autoregressive video generation