Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot

In plain English

The authors built a system that coordinates speech, audio-driven facial animation and body gesture on a physical humanoid in real time. Rather than generating motion end to end, it assembles gestures from a library of clips cut from the BEAT2 motion dataset, each checked for clean boundaries and communicative function, with a planner sequencing them to match the length and meaning of each spoken reply. In the authors' own comparison it rated highest on naturalness, expressiveness and multimodal coherence. The practitioner takeaway is that a pre-vetted gesture library plus a planner is a cheaper and more controllable route to coherent body language than a generative motion model.