TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation

2026-03-06

AI Summary

This work addresses the challenges of high inference latency, temporal instability (e.g., flickering and identity drift), and audio-visual asynchrony under complex speech inputs in diffusion-based talking-head generation. To overcome these issues, the authors propose a reference-conditioned latent diffusion framework based on teacher–student distillation. Temporal consistency is enhanced through identity anchoring and temporal regularization, while coarse-grained lip motion is controlled via phoneme-to-viseme mapping. Notably, this approach is the first to integrate distillation with explicit temporal consistency constraints. Evaluated on the LRS3 dataset, the method achieves reconstruction quality comparable to that of the teacher model using only a minimal number of diffusion steps, substantially reducing inference latency and demonstrating strong feasibility for deployment on edge devices.

Technology Category

Computer Vision: Diffusion Models for VisionMachine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Generation

Application Category

Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsEconomics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Diffusion models have recently advanced photorealistic human synthesis, although practical talking-head generation (THG) remains constrained by high inference latency, temporal instability such as flicker and identity drift, and imperfect audio-visual alignment under challenging speech conditions. This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework that explores few-step inference for efficient audio-driven talking-head generation. The approach adopts a teacher-student distillation formulation in which a diffusion teacher trained with a standard noise prediction objective guides a lightweight student denoiser capable of operating with significantly fewer inference steps to improve generation stability. The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker during synthesis, while viseme-based audio conditioning provides coarse lip motion control. Experiments on the LRS3 dataset report denoising-stage component-level metrics relative to VAE reconstructions and preliminary latency characterization, including CPU-only and edge computing measurements and feasibility estimates for edge deployment. The results suggest that distilled diffusion models can retain much of the reconstruction behaviour of a stronger teacher while enabling substantially lower latency inference. The study is positioned as an initial step toward practical diffusion-based talking-head generation under constrained computational settings. GitHub: https://mazumdarsoumya.github.io/TempoSyncDiff

Problem

• talking-head generation
• diffusion models
• temporal consistency
• audio-visual alignment
• inference latency

Innovation

• distilled diffusion
• temporal consistency
• low-latency talking head
• identity anchoring
• audio-visual alignment