Embodied Referring Expression Comprehension in Human-Robot Interaction

2025-12-06

AI Summary

Existing HRI datasets suffer from limited viewpoint diversity, insufficient coverage of nonverbal gestures, and imbalanced indoor-outdoor scene distributions, hindering embodied referential understanding. To address these limitations, we introduce Refer360—the first large-scale, multi-view, indoor-outdoor-coordinated HRI dataset featuring aligned instruction-gesture pairs. We further propose MuRes, a multimodal guided residual module that achieves deep fusion of visual, linguistic, and pose modalities through modality-specific feature extraction, information bottleneck regularization, and residual enhancement. Extensive experiments across four benchmark datasets demonstrate that MuRes consistently improves model accuracy by an average of +4.2%, while exhibiting strong cross-dataset generalization. Collectively, Refer360 and MuRes establish a scalable, embodied comprehension framework for natural human-robot collaboration in real-world settings.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.

Problem

• Addresses lack of large-scale embodied interaction datasets in HRI
• Overcomes perspective bias and limited nonverbal gesture coverage
• Improves robot comprehension of embodied human instructions

Innovation

• Large-scale multimodal dataset with diverse viewpoints
• Guided residual module for complementary feature fusion
• Information bottleneck for salient modality-specific signals