Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization

2026-06-09

AI Summary

Existing diffusion models struggle to achieve real-time lip synchronization due to their reliance on full-sequence bidirectional attention and multi-step denoising. This work proposes Lip Forcing, the first approach to introduce autoregressive diffusion mechanisms to video-to-video lip-sync tasks. By distilling a causal student model from a 14B-parameter bidirectional teacher, the method enables real-time generation with only two denoising steps and without classifier-free guidance (CFG) during inference. The study reveals a fidelity–synchronization trade-off inherent in CFG and accordingly designs a Sync-Window DMD framework, a two-step scheduler, and a SyncNet-based reward mechanism. The resulting 1.3B-parameter student model achieves 31 FPS—17.6× faster than comparable bidirectional models—while the distilled 14B model attains a 39.8× speedup with comparable fidelity and sub-millisecond initial-frame latency, significantly outperforming existing diffusion baselines.

Abstract

Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.

Problem

• lip synchronization
• real-time inference
• diffusion models
• video-to-video generation
• autoregressive modeling

Innovation

• autoregressive diffusion
• lip synchronization
• model distillation
• real-time inference
• video-to-video generation