[Paper] Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
Source: arXiv - 2606.11180v1
Overview
Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.
Key Contributions
This paper presents research in the following areas:
- cs.CV
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.CV.
Authors
- Paul Hyunbin Cho
- Jinhyuk Jang
- SeokYoung Lee
- Joungbin Lee
- Siyoon Jin
- Heeseong Shin
- Jung Yi
- Yunjin Park
- Chulmin Park
- Seungryong Kim
Paper Information
- arXiv ID: 2606.11180v1
- Categories: cs.CV
- Published: June 9, 2026
- PDF: Download PDF