PVSync: A Unified Lip-Sync Expert for Timing and Articulation

2026-10-06

AI Summary

This study addresses the limitation of existing lip-sync evaluation metrics, which match audio-visual temporal alignment while neglecting articulatory content. We propose the first unified framework integrating temporal offset estimation with pronunciation scoring. The method aligns audio-visual embeddings by combining window-level contrastive learning with a viseme-level pronunciation objective, and leverages forced alignment to automatically generate viseme labels from transcripts without manual annotation. Experimental results demonstrate that the proposed model achieves a Spearman correlation coefficient of 0.83 across 13 generative models—substantially outperforming LSE-C at 0.34—and yields a viseme discrimination ROC AUC of 0.91. These findings indicate comprehensive superiority over baseline methods such as SyncNet, establishing a more robust standard for evaluating generated talking-face videos.

Abstract

Lip movements can match the timing of speech without matching the spoken sounds. We introduce PVSync, a unified model for audio-visual offset estimation and phoneme-level articulation scoring. PVSync combines window-level contrastive learning for synchronisation with a phoneme-level articulation objective that aligns audio and video embeddings of the same viseme class across clips. Visemes group phonemes with similar visible articulation. Viseme labels are derived automatically from forced-aligned transcripts, without manual annotations. On offset-corrected videos from 13 talking-head video generation models, PVSync matches human rankings of lip-sync quality more closely than LSE-C, achieving a Spearman correlation of 0.83 versus 0.34. On an automatically constructed benchmark from held-out speech, PVSync distinguishes viseme-matched from mismatched audio-visual pairs with an ROC AUC of 0.91. PVSync also outperforms SyncNet and MTD-VocaLiST in temporal offset recovery on held-out in-the-wild clips. Code and benchmark data will be released upon acceptance.

Problem

• lip-sync evaluation
• audio-visual synchronization
• phoneme-level articulation
• temporal offset estimation
• talking-head generation

Innovation

• Lip-Sync
• Viseme-level Articulation
• Contrastive Learning
• Audio-Visual Offset Estimation
• Talking-head Generation