Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming

2026-08-06

AI Summary

This work addresses the challenges of error accumulation and imprecise speech-content alignment in causal synthesis for real-time, long-form digital human generation. To overcome these issues, the authors propose a post-training framework that trains a causal generator using a hybrid of teacher forcing and diffusion forcing, augmented with long-range self-forcing and DMD distillation. Crucially, an external language model predicts 25 Hz speech-planning tokens to precisely control speech progression. This approach represents the first effective adaptation of pretrained bidirectional models to real-time streaming text-to-audiovisual (T2AV) generation, achieving accurate audiovisual synchronization without exposure bias. Experimental results demonstrate that the system operates at 27.12 FPS, delivering high-quality lip-sync and identity consistency while exceeding the 24 FPS requirement for real-time playback.

Abstract

Real-time long-form avatar audio--video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual consistency. Adapting a pretrained bidirectional model to this setting presents two key dilemmas. First, autoregressively reusing generated blocks as context creates exposure bias, causing errors and visual drift to accumulate over long rollouts. Second, a global speech utterance does not indicates a causal generator which portion should be spoken next when only limited local audio--video context is available. We present \textbf{Vorch-Streamer}, a post-training framework that addresses these challenges and enables real-time long-form Text-to-Audio-Video (T2AV) streaming. We construct a synthetic corpus of 80K avatar clips spanning 12--21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing. We then apply long-horizon Self Forcing with DMD distillation, exposing the model to its own rollout distribution while preserving the quality of the pretrained bidirectional teacher. To explicitly control speech progression, an external language model predicts discrete 25-Hz speech-planning tokens, whose continuous features condition the audio diffusion branch and align each causal block with the content it should speak. With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate while maintaining competitive audio--lip synchronization and strong identity preservation over long-form generation.

Problem

• real-time
• long-form
• audio-visual generation
• causal synthesis
• avatar streaming

Innovation

• causal generation
• exposure bias mitigation
• speech-planning tokens
• real-time T2AV streaming
• DMD distillation