STEER: Steerable Dyadic Head Avatars

2026-07-26

AI Summary

This work addresses the challenge of explicitly controlling nonverbal behaviors—such as gaze, head motion rhythm, and emotional expression—in speech-driven facial animation during dyadic conversations. We propose the first 3D talking-head prior model that enables fine-grained, explicit control over these behaviors by disentangling their underlying signals, thereby allowing precise manipulation of a virtual avatar’s listening, reactive, and interactive actions. To facilitate training, we introduce an automated pipeline for extracting pseudo-labels of nonverbal behaviors from in-the-wild conversational videos. Our approach employs a causal flow-matching Transformer to model the influence of audio, interlocutor motion, and user-specified control signals on target head dynamics, integrated with a Gaussian head avatar prior to achieve high-fidelity, real-time animation without retraining. Experiments demonstrate superior performance over existing dyadic motion baselines in terms of motion quality, expressiveness, and diversity. Code and dataset are publicly released.

Abstract

Facial movement and expression are central to face-to-face communication, conveying turn-taking, attention, agreement, and engagement alongside speech. While speech-driven facial animation has made strong progress in lip synchronization and audio-conditioned motion generation, most methods treat conversational behavior as an emergent byproduct of audio, or expose only coarse sequence-level affect control. As a result, key non-verbal channels such as gaze contact and aversion, rhythmic head motion, and emotion remain difficult to explicitly control. We present STEER, a controllable 3D dyadic motion prior for reactive conversational head avatars. STEER factorizes conversational behavior into explicit controls for gaze, head rhythm, and emotion, allowing users to steer how an avatar listens, reacts, and engages with a conversation partner. Since temporally aligned annotations for these behaviors are not available in public dyadic corpora, we introduce a tracking and annotation pipeline that recovers behavioral pseudo-labels from in-the-wild dyadic video. A causal flow-matching transformer then learns partner-aware target motion conditioned on audio, partner motion, emotion and the proposed behavioral controls. We further embed STEER in a photorealistic avatar pipeline by extending a Universal Gaussian Head-Avatar Prior with a learned mapping from tracked parametric motion into its avatar-driving space. This enables controllable animation of high-fidelity Gaussian head avatars without re-training the underlying avatar model. STEER outperforms recent dyadic motion baselines on motion quality, dynamics, and diversity, remains competitive on partner coupling, and enables gaze, head-rhythm, and emotion edits together with an interactive live deployment. We make our code and dataset annotations available at our webpage.

Problem

• facial animation
• non-verbal behavior
• gaze control
• head motion
• emotion expression

Innovation

• steerable avatars
• dyadic conversation
• behavioral control
• flow-matching transformer
• Gaussian head avatar