GIVE: Grounding Human Gestures in Vision-Language-Action Models

2026-06-11

AI Summary

This work addresses the limitation of existing vision-language-action (VLA) models in human-robot interaction, which often ignore gestural cues, leading to ambiguous intent interpretation and unreliable execution when verbal instructions are imprecise. The authors propose the first approach to seamlessly integrate human gestures into a pretrained VLA model without architectural modifications. Specifically, the visual pathway explicitly localizes target objects using hand skeletons and fingertip rays, while the semantic pathway generates high-level descriptions linking gestures to task context to enhance intent understanding. Through multimodal fusion and policy fine-tuning, the method achieves a 40% improvement in object identification accuracy and an 80% increase in task success rate in real-world interactive experiments, demonstrating strong generalization across novel scenarios and diverse users.

Abstract

Human communication is inherently multimodal, where language is often accompanied by non-verbal cues such as gestures to convey intentions. However, current Vision-Language-Action (VLA) models treat robotic manipulation as a pure text-driven task, overlooking the important role of gestures in Human-Robot Interaction (HRI). This often leads to inaccurate intent grounding and unreliable manipulation when language instructions are ambiguous or underspecified. To address this challenge, we propose GIVE (Gesture Intent via Visual-Semantic Enhancement), an effective approach that enhances pre-trained VLA models with human gesture understanding without architectural modifications. Specifically, GIVE incorporates gesture information through two complementary pathways: a visual pathway that overlays hand skeletons and fingertip rays onto robot observations for explicit object grounding, and a semantic pathway that generates high-level descriptions of human gestures and task instructions for robust intent grounding. By jointly leveraging visual and semantic guidance, GIVE enables VLA policies to better associate gestures with manipulation behaviors and adapt to dynamic interaction intents. In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%, while demonstrating strong robustness and generalization to unseen spatial layouts and diverse participants.

Problem

• Human-Robot Interaction
• Gesture Understanding
• Vision-Language-Action Models
• Intent Grounding
• Multimodal Communication

Innovation

• gesture grounding
• vision-language-action models
• human-robot interaction
• multimodal intent understanding
• visual-semantic enhancement