Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation

In plain English

Hallo-Live generates the portrait video and the speech together from text rather than animating a face to pre-recorded audio, and most of the paper is about making that fast enough to hold a conversation with. Letting each video block attend to a short window of upcoming audio keeps lip motion ahead of the speech without breaking streaming causality, and reweighting the distillation with human-preference reward models limits the usual quality loss from few-step acceleration: the authors report 20.4 FPS at 0.94 seconds latency on two H200 GPUs, around 16x the throughput and a hundredth of the latency of the teacher model, at comparable quality and sync scores. Joint audio-video generation is the architecture to watch if you want an avatar whose voice and face come from one model instead of a stitched pipeline; all figures are the authors' own.