[Paper] The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?
Source: arXiv - 2606.07435v1
Overview
Visual speech recognition (VSR) models now surpass human lipreaders on benchmarks, but do such gains establish human-like visual speech perception? To explore this, we compare three VSR systems with human baselines on the MaFI word-level lipreading dataset using word, character, phoneme, and viseme-level metrics. Although models achieve higher overall accuracy, they succeed and fail on different words than humans. A text-only n-gram baseline given only a few initial phonemes rivals human lipreading. VSR word-level errors are consistently better explained by training word frequency than by the visual informativeness of words. Viseme accuracies, confusion matrices and human-model correlations further show that models gain most on visemes humans find hardest, and show much weaker dependence on visual clarity. Our work demonstrates that VSR systems rely primarily on language cues from training data rather than visual perception, failing to bind visual features into meaningful words.
Key Contributions
This paper presents research in the following areas:
- cs.CV
- cs.CL
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.CV.
Authors
- Rishabh Jain
- Naomi Harte
Paper Information
- arXiv ID: 2606.07435v1
- Categories: cs.CV, cs.CL
- Published: June 5, 2026
- PDF: Download PDF