ETHead: Generating Expressive 3D Facial Animation and Head Movement from Speech

2026-08-02

AI Summary

Existing methods struggle to model the complex emotional dynamics of speech-driven 3D talking heads due to the scarcity of high-fidelity 3D data. To address this, this work proposes a self-distillation framework that leverages large-scale 2D talking videos to pretrain a speech encoder and introduces a probabilistic emotion-modulated masking mechanism to align expressive visual dynamics within a shared latent space between speech and motion. Without requiring extensive 3D training data, the approach generates highly expressive 3D facial and head animations, significantly outperforming state-of-the-art methods across multiple evaluation metrics. Moreover, the learned speech encoder demonstrates strong transferability, effectively enhancing the expressiveness of other 3D talking-head systems when integrated.

Abstract

Generating expressive 3D talking heads solely from speech remains a significant challenge due to the scarcity of high-fidelity 3D data, which limits the modeling of complex emotional motion patterns. In this paper, we introduce \textbf{E}xpressive \textbf{T}alking \textbf{Head} (ETHead), a method for generating 3D facial and head motions that vividly align with the emotional content of input speech. To overcome the data limitations, we design a self-distillation framework that leverages large-scale 2D talking videos to pre-train a specialized speech encoder. By incorporating a novel emotion-modulated probabilistic masking mechanism, this framework aligns speech representations with expressive visual dynamics, allowing the encoder to extract features highly correlated with facial and head motions directly from audio. These features are then leveraged to guide 3D generation, enriching input cues and providing explicit supervision through a joint speech-motion latent space. Extensive experiments demonstrate that ETHead substantially outperforms state-of-the-art methods. Furthermore, our motion-aligned speech encoder can serve as a transferable module, offering a general solution for enhancing expressiveness in other 3D talking head animation frameworks. The project page is available at https://verdure-oss.github.io/ETHead.github.io/.

Problem

• expressive 3D facial animation
• head movement
• speech-driven animation
• emotional motion patterns
• 3D talking head

Innovation

• expressive 3D talking head
• speech-driven animation
• self-distillation
• emotion-modulated masking
• joint speech-motion latent space