preprint · arXiv
EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents
Yu Zhang, Kaiyuan Shen, Yang Li
In plain English
A talking-head system that gives an LLM a real-time face and, unlike most, treats listening as a first-class state: it suppresses mouth movement while the user is talking and switches on turn-taking, without needing to hear the other party's audio in advance. The authors report 59 FPS on a single GPU using four sampling steps, trained on 27.2 hours of curated video. For anyone building live conversational avatars, the causal single-stream design is the part that matters — it is what lets an avatar work in a genuine two-way call rather than only in scripted playback.
Abstract
We present EmbodiedHead, a speech-driven talking-head framework that equips LLMs with real-time visual avatars for conversation. A practical embodied avatar must achieve real-time generation, unified listening-speaking behavior, and high rendered visual quality simultaneously. Our framework couples the first Rectified-Flow Diffusion Transformer (DiT) for this task with a differentiable renderer, enabling diverse, high-fidelity generation in as few as four sampling steps. Prior listening-speaking methods rely on dual-stream audio, introducing an interlocutor look-ahead dependency incompatible with causal user-LLM interaction. We instead adopt a single-stream interface with explicit per-frame listening-speaking state conditioning and a Streaming Audio Scheduler, suppressing spurious mouth motion during listening while enabling seamless turn-taking. A two-stage training scheme of coefficient-space pretraining and joint image-domain refinement further closes the gap between motion-level supervision and rendered quality. Extensive experiments demonstrate state-of-the-art visual quality and motion fidelity in both speaking and listening scenarios.