preprint · arXiv
ComplexSync: High-Fidelity and Real-Time Lip Sync in Complex Scenarios
Jiaran Cai, Xingpei Ma, Shenneng Huang
In plain English
Most lip-sync models are benchmarked on clean, front-facing footage and degrade badly on real video with occlusions, moving backgrounds and changing light. This work trains two parallel streams — one for audio-visual alignment, one for preserving identity and background — distils the model down to a single denoising step, and adds a feature-relationship loss to handle occlusion. The authors report 72.73 FPS and the best scores on their own 200-video benchmark of difficult conditions, plus a 50-person preference study. All figures are self-reported in a preprint, but the framing is the useful part: the gap between benchmark lip-sync quality and messy production footage is where most deployments actually fail.