A Benchmark for Spatially Grounded Gesture Generation

2026-10-02

AI Summary

This study addresses the inability of existing gesture generation metrics to disentangle naturalness from spatial pointing accuracy. To this end, we introduce MM-Conv-Flow, the first spatially anchored gesture benchmark comprising 2K VR dialogue segments, and propose a multidimensional evaluation protocol that decouples spatiotemporal alignment, spatial localization, and naturalness. Furthermore, we develop a flow matching-based baseline model for this task. Experimental results demonstrate that the proposed model surpasses human performance in geometric localization accuracy while maintaining spatial precision independent of perceived naturalness. This work substantiates the necessity of multidimensional, independent evaluation and establishes a new standard for assessing the quality of generated gestures.

Abstract

Communication in shared space interweaves verbal and non-verbal signals, and pointing gestures anchor language to the environment: "put the cup on that one" is uninterpretable without the gesture that fixes the referent. Yet no common framework exists for evaluating whether generated gestures indicate their intended referent; distributional metrics reward a gesture aimed at the wrong object as long as it looks natural. We introduce a benchmark for spatially grounded gesture generation, comprising ~2K pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents, a task in which systems must decide when, how and where to point within conversational speech, and a protocol that separates temporal alignment, spatial grounding and perceived naturalness. We also provide a flow-matching baseline, MM-Conv-Flow. Evaluating it alongside an independent retrieval-based system and captured human motion, we find that geometric grounding can exceed that of human pointing without any gain in perceived naturalness, showing that referential gesture quality must be measured along separate dimensions.

Problem

• gesture generation
• spatial grounding
• benchmark
• pointing gestures
• evaluation metrics

Innovation

• spatially grounded gesture generation
• benchmark
• flow matching
• evaluation protocol
• VR dialogue