When Less Is More: A Sparse Facial Motion Structure For Listening Motion Learning

2025-04-08

AI Summary

Current methods for listener head motion prediction in dyadic conversations rely on discrete motion tokens derived from continuous signals, which struggle to capture the temporal dynamics and individual variability of nonverbal facial movements—resulting in low generation fidelity and poor interpretability. To address this, we propose a sparse structured facial motion representation: semantic-meaningful keyframes encode salient poses, while interpolated transition frames model motion evolution—replacing conventional discrete token sequences. This formulation avoids pattern distortion induced by continuous-to-discrete quantization and enables efficient modeling within a self-supervised listener head prediction framework. Experiments demonstrate that our approach outperforms state-of-the-art methods across motion quality (FID, MSE), diversity (APD), and interpretability, significantly advancing nonverbal behavior modeling.

Technology Category

• Computer Vision: Motion & Tracking
• Natural Language Processing: Generation
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Effective human behavior modeling is critical for successful human-robot interaction. Current state-of-the-art approaches for predicting listening head behavior during dyadic conversations employ continuous-to-discrete representations, where continuous facial motion sequence is converted into discrete latent tokens. However, non-verbal facial motion presents unique challenges owing to its temporal variance and multi-modal nature. State-of-the-art discrete motion token representation struggles to capture underlying non-verbal facial patterns making training the listening head inefficient with low-fidelity generated motion. This study proposes a novel method for representing and predicting non-verbal facial motion by encoding long sequences into a sparse sequence of keyframes and transition frames. By identifying crucial motion steps and interpolating intermediate frames, our method preserves the temporal structure of motion while enhancing instance-wise diversity during the learning process. Additionally, we apply this novel sparse representation to the task of listening head prediction, demonstrating its contribution to improving the explanation of facial motion patterns.

Problem

• Predicting listening head behavior in human-robot interaction
• Capturing non-verbal facial motion patterns efficiently
• Improving fidelity of generated facial motion sequences

Innovation

• Sparse keyframe and transition frame encoding
• Interpolating intermediate frames for diversity
• Improved facial motion pattern explanation