The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?

2026-06-05

AI Summary

This study investigates whether current visual speech recognition (VSR) models possess human-like visual speech perception capabilities in lip reading. By comparing model performance against human lip readers across four granularities—words, characters, phonemes, and visemes—on the MaFI dataset, and integrating confusion matrices, human–model correlation analyses, and n-gram language baselines, the work reveals that VSR models predominantly rely on linguistic priors rather than genuine visual cues. Although models achieve higher overall accuracy, their error patterns markedly diverge from those of humans, with performance more strongly influenced by word frequency than by visual distinctiveness. Notably, models show the greatest gains on visemes that are most challenging for humans, yet exhibit insufficient utilization of visual information. This challenges the prevailing paradigm of evaluating perceptual competence solely through accuracy and underscores the importance of multi-granularity behavioral alignment.

Abstract

Visual speech recognition (VSR) models now surpass human lipreaders on benchmarks, but do such gains establish human-like visual speech perception? To explore this, we compare three VSR systems with human baselines on the MaFI word-level lipreading dataset using word, character, phoneme, and viseme-level metrics. Although models achieve higher overall accuracy, they succeed and fail on different words than humans. A text-only n-gram baseline given only a few initial phonemes rivals human lipreading. VSR word-level errors are consistently better explained by training word frequency than by the visual informativeness of words. Viseme accuracies, confusion matrices and human-model correlations further show that models gain most on visemes humans find hardest, and show much weaker dependence on visual clarity. Our work demonstrates that VSR systems rely primarily on language cues from training data rather than visual perception, failing to bind visual features into meaningful words.

Problem

• visual speech recognition
• lipreading
• human perception
• viseme
• language cues

Innovation

• visual speech recognition
• lipreading
• viseme
• language priors
• human-model comparison