VarGes: Improving Variation in Co-Speech 3D Gesture Generation via StyleCLIPS

2025-02-15

AI Summary

Existing speech-driven 3D gesture generation methods suffer from weak audio representation capability and insufficient training data diversity, resulting in monotonous and unnatural gestures. To address this, we propose StyleCLIPS—the first representation framework explicitly designed for gesture style modeling—comprising three synergistic modules: Visual Enhancement Feature Extraction (VEFE), Voice–Style Coupling Encoding (VCSE), and Pose-Decoupled Generation (VDGP). Our approach jointly models audio, style, and pose across modalities by integrating MFCC-based audio features, 3D pose estimation, StyleCLIPS visual style embeddings, additive attention-pooling Transformers, and a cross-conditioned autoregressive generator. Evaluated on standard benchmarks, our method achieves a 18.7% reduction in Frechet Video Distance (FVD) and a 12.3% reduction in Mel Cepstral Distortion (MCD), with both quantitative metrics and qualitative analysis confirming substantial improvements in gesture diversity and naturalness.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Generating expressive and diverse human gestures from audio is crucial in fields like human-computer interaction, virtual reality, and animation. Though existing methods have achieved remarkable performance, they often exhibit limitations due to constrained dataset diversity and the restricted amount of information derived from audio inputs. To address these challenges, we present VarGes, a novel variation-driven framework designed to enhance co-speech gesture generation by integrating visual stylistic cues while maintaining naturalness. Our approach begins with the Variation-Enhanced Feature Extraction (VEFE) module, which seamlessly incorporates extcolor{blue}{style-reference} video data into a 3D human pose estimation network to extract StyleCLIPS, thereby enriching the input with stylistic information. Subsequently, we employ the Variation-Compensation Style Encoder (VCSE), a transformer-style encoder equipped with an additive attention mechanism pooling layer, to robustly encode diverse StyleCLIPS representations and effectively manage stylistic variations. Finally, the Variation-Driven Gesture Predictor (VDGP) module fuses MFCC audio features with StyleCLIPS encodings via cross-attention, injecting this fused data into a cross-conditional autoregressive model to modulate 3D human gesture generation based on audio input and stylistic clues. The efficacy of our approach is validated on benchmark datasets, where it outperforms existing methods in terms of gesture diversity and naturalness. The code and video results will be made publicly available upon acceptance:https://github.com/mookerr/VarGES/ .

Problem

• Enhancing co-speech 3D gesture generation
• Integrating visual stylistic cues
• Improving gesture diversity and naturalness

Innovation

• Integrates StyleCLIPS for visual cues
• Uses transformer-style encoder for style management
• Fuses audio and style via cross-attention