Vorch-Director: Interactive World Story Model via Noise-Aware Error Rectification

2026-08-06

AI Summary

This work addresses identity drift, excessive smoothing, and audio-visual desynchronization in autoregressive audio-visual generation—issues primarily caused by error accumulation. To mitigate these problems, the authors propose a noise-aware residual correction mechanism that dynamically links prediction residuals to the noise level present during generation. During training, errors corresponding to specific noise stages are injected to enhance the fidelity of historical states. Integrated with task embeddings and clean conditional pooling, the approach enables long-duration generation across multiple shots, subjects, and reference-guided scenarios. Built upon an audio-visual LTX-2 diffusion Transformer and leveraging flow matching for denoising alongside mixed-task training, the method significantly outperforms strong baselines on both ST-Bench and a newly introduced long-sequence benchmark, achieving notable improvements in generation quality, stability, and audio-visual synchronization.

Abstract

Autoregressive continuation provides a natural path toward minute-scale audio-visual generation by repeatedly extending a short-window generator conditioned on previously generated video and audio. However, models are trained on clean ground-truth histories, while inference relies on their own generated histories, where accumulated errors cause identity drift, over-smoothing, and audio-visual desynchronization. Recent methods reduce this mismatch by reusing prediction residuals as synthetic corruption, but we observe that the effectiveness of residual correction critically depends on the flow-matching noise level at which residuals are produced. We propose Vorch-Director, a noise-level-aware residual correction strategy that associates each residual with its originating noise level and injects residuals from matched noise regimes during training. By aligning injected errors with the denoising process, Vorch-Director produces more realistic autoregressive histories while retaining efficient teacher-forcing training. Built on the audio-visual LTX-2 diffusion transformer, Vorch-Director further introduces task embeddings to distinguish historical video, reference images, and target video, enabling unified conditioning for long-horizon generation. Together with a clean conditioning sink and mixed-task training, Vorch-Director supports multi-shot, multi-subject, reference-guided audio-visual long-video generation. We evaluate Vorch-Director on ST-Bench and introduce a new long-horizon audio-visual benchmark with metrics for quality drift and long-range consistency. Extensive experiments demonstrate improved stability and audio-visual fidelity over strong baselines.

Problem

• autoregressive generation
• error accumulation
• audio-visual desynchronization
• identity drift
• over-smoothing

Innovation

• noise-level-aware residual correction
• autoregressive audio-visual generation
• diffusion transformer
• task embeddings
• long-horizon video synthesis