JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents

In plain English

Conversational agents usually generate speech and face separately, or skip the face altogether. This system runs both from one shared LLM through a joint audio-visual representation that then splits into streaming speech and face branches, removing the intermediate hand-off. The authors report initial latency roughly 200 times lower than a strong baseline, though no absolute millisecond figures, plus better lip sync and more appropriate expressions. They also release JAVIS-Syn, a 334-hour synthetic dataset of expressive multi-turn conversations, which addresses a real shortage of training data for audio-visual dialogue.