Full-Duplex Interaction with Speech, Expressions and Gestures

In plain English

ELLSA 2, from Tsinghua and ByteDance, generates speech, facial expression and gesture together in 0.8-second chunks, and keeps generating while it is listening rather than freezing between turns. It reports 95.2% average turn-taking accuracy and the best spoken-answer accuracy of the systems compared, and gets semantic gestures by retrieving an exemplar from a 3,227-item library when a learned head decides a gesture is warranted. The retrieval trick is the practical takeaway: it raised human-rated gesture meaningfulness from 2.01 to 2.71 without a bigger model.