preprint · arXiv
EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence
Yahui Li, Yinfeng Yu, Liejun Wang, Shengjie Shen
In plain English
Emotional avatar video usually forces a trade: add richer emotional control and lip-sync gets worse. This method uses an LLM to turn real videos into descriptive emotional guidance instead of coarse labels like "happy", plus two mechanisms aimed at keeping long videos coherent frame to frame. It reports gains on lip-sync, temporal consistency and emotional accuracy together on standard datasets — which is the combination practitioners actually need, since drift over a long take is what breaks avatar content in production.
Abstract
Emotional talking head video generation aims to generate expressive portrait videos with accurate lip synchronization and emotional facial expressions. Current methods rely on simple emotional labels, leading to insufficient semantic information. While introducing high-level semantics enhances expressiveness, it easily causes lip-sync degradation. Furthermore, mainstream generation methods struggle to balance computational efficiency and global motion awareness in long videos, and suffer from poor temporal coherence. Therefore, we propose an Emotion-Aware Diffusion model-based Network, called EAD-Net. We introduce SyncNet supervision and Temporal Representation Alignment (TREPA) to mitigate lip-sync degradation caused by multi-modal fusion. To model complex spatio-temporal dependencies in long video sequences, we propose a Spatio-Temporal Directional Attention (STDA) mechanism that captures global motion patterns through strip attention. Additionally, we design a Temporal Frame graph Reasoning Module (TFRM) to explicitly model temporal coherence between video frames through graph structure learning. To enhance emotional semantic control, a large language model is employed to extract textual descriptions from real videos, serving as high-level semantic guidance. Experiments on the HDTF and MEAD datasets demonstrate that our method outperforms existing methods in terms of lip-sync accuracy, temporal consistency and emotional accuracy.