LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

2026-07-29

AI Summary

This work addresses the trade-off between latency and quality in long-duration, real-time digital human video generation, where multi-step diffusion models hinder streaming inference and autoregressive approaches suffer from error accumulation and identity drift. To overcome these limitations, the authors propose LeapTalk, a novel framework that introduces, for the first time, a data-to-data Brownian bridge transport paradigm to talking-head synthesis. LeapTalk integrates single-step bridge distillation, heterogeneous knowledge distillation, and SNR-aligned temporal transformation, along with an audio-driven classifier-free guidance mechanism to effectively mitigate identity drift and enhance lip-sync accuracy. Requiring only a single forward pass, the method enables streaming generation of arbitrarily long, high-fidelity, and temporally coherent videos at 200 FPS, significantly outperforming existing approaches in efficiency, stability, and output quality.

Abstract

Long-form and real-time talking-head generation remains challenging due to a latency-quality trade-off: inefficient multi-step diffusion prohibits streaming generation, whereas real-time autoregressive approaches suffer from error accumulation and identity drift. To address this drawback, we propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos. At the heart of our approach lies a single-step bridge distillation scheme. On the one hand, departing from the conventional noise-to-data paradigm, we introduce a data-to-data transport formulation based on a Brownian bridge. Anchored by a persistent reference, this strategy effectively mitigates identity drift and enhances long-term temporal stability. On the other hand, to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model, we explore a heterogeneous distillation framework with an SNR-aligned time transformation $Φ(τ)$, which bridges the functional discrepancy between the two models. Moreover, we propose an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction. Extensive experiments demonstrate that our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS, significantly outperforming existing approaches in both efficiency and stability. Project Page: https://zhangrongxiang.github.io/leaptalk-page/

Problem

• latency-quality trade-off
• talking head generation
• identity drift
• real-time generation
• temporal stability

Innovation

• single-step generation
• Brownian bridge
• heterogeneous distillation
• audio-driven guidance
• talking head synthesis