Recognizing Co-Speech Gestures in-the-Wild

2026-05-29

AI Summary

Existing co-speech gesture recognition methods are hindered by the scarcity of large-scale, finely annotated real-world data, making it difficult to accurately model the semantic correspondence between gestures and spoken words. To address this limitation, this work introduces GRW, the first large-scale co-speech gesture dataset collected in unconstrained environments, comprising 156,688 video clips with frame-level temporal annotations across 150 lexical categories. Leveraging this dataset, we establish the first benchmark for co-speech gesture understanding centered on three core tasks: gesture semantic classification, corresponding word recognition, and temporal localization. Experimental results demonstrate that the proposed benchmark substantially advances fine-grained comprehension and precise temporal grounding of co-speech gestures.

Abstract

While humans naturally gesture during speech, only a sparse subset of these movements are visually depictive and semantically linked to specific spoken words. Current multimodal models struggle to capture these semantic co-speech gestures, heavily bottlenecked by a lack of precisely annotated training data. To address this, we introduce the Gesture Recognition in the Wild (GRW) dataset, the first large-scale benchmark designed to map unconstrained human gestures to specific words with frame-accurate temporal boundaries. Comprising 156,688 manually annotated video clips, GRW spans a highly diverse 150-word taxonomy of physical actions, spatial descriptors, and abstract concepts. We leverage GRW to train video models to (a) classify gestures as semantic or not, (b) recognize the word corresponding to a co-speech gesture, and (c) temporally localize the gesture. We also use GRW to establish benchmarks for these three tasks.

Problem

• co-speech gestures
• semantic gestures
• gesture recognition
• multimodal learning
• temporal localization

Innovation

• co-speech gestures
• semantic gesture recognition
• temporal localization
• multimodal learning
• GRW dataset