StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

2026-08-02

AI Summary

This work addresses the issue of error accumulation and long-term pose drift prevalent in existing open-loop streaming speech-driven gesture generation methods. To mitigate these limitations, the authors propose a closed-loop streaming framework that periodically executes a generate–retrieve–refine cycle, anchoring motion trajectories at segment boundaries with key pose anchors. The approach innovatively integrates a key-pose anchoring mechanism, a stochastic anchor masking training strategy, and a part-aware DiT architecture that disentangles hand, body, and locomotion motion streams. This design effectively suppresses temporal drift while preserving natural gesture dynamics. Evaluated on the BEAT2 dataset, the method achieves state-of-the-art performance in terms of the FGD metric and maintains real-time efficiency at 76 frames per second.

Abstract

Real-time co-speech gesture generation must produce 3D motion clip by clip as speech arrives. Existing streaming methods are open-loop: each clip depends on past context, but the model cannot check or correct its trajectory. Small errors therefore accumulate and cause drift over long sequences. We observe that this failure is mainly caused by the lack of a forward constraint rather than poor short-clip quality. A plausible key pose at the end of each clip provides a destination anchor that limits drift. Based on this observation, we propose StreamTalk, a closed-loop framework with a periodic generate-retrieve-refine cycle. Streaming Pose-Guided Generation first predicts a coarse clip, retrieves a plausible tail pose from a speaker-specific motion database, and refines the clip using this pose before continuing to the next window. During training, Stochastic Anchor Masking randomly masks pose and translation frames, teaching the model to recover complete motion from sparse boundary conditions. A part-aware DiT separates hand, body, and translation streams to reduce interference between global displacement and local articulation. On BEAT2, StreamTalk achieves state-of-the-art FGD, reduces long-horizon drift relative to open-loop baselines, and runs in real time at 76 FPS. Project page: https://xiangyue-zhang.github.io/StreamTalk/.

Problem

• co-speech gesture generation
• streaming animation
• motion drift
• real-time generation
• key-pose anchoring

Innovation

• key-pose anchoring
• closed-loop generation
• streaming co-speech gesture
• pose-guided refinement
• part-aware DiT