MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning

2026-09-21

AI Summary

本文提出MIGU框架，通过结合语义和几何证据解决多模态指令在不确定性下的理解问题，以支持操作规划。

Abstract

Understanding natural human instructions is crucial for deploying robots in human-centric environments. We study multimodal instruction grounding, where language and gesture provide complementary but uncertain cues. We present MIGU, a modular framework that combines semantic and geometric evidence into a unified grounding belief and connects it to manipulation planning. MIGU constructs a 3D geometric likelihood by propagating viewing-direction and depth uncertainty through eye-finger geometry while accounting for hand-direction estimation error. A vision-language model (VLM) provides semantic priors over candidate objects and regions, which are combined with the geometric likelihood through Bayes-inspired fusion. The resulting belief supports behavior planning to either proceed directly to downstream planning or request clarification. Grounded targets then define goals for mobile manipulation and tabletop task-and-motion planning. On a real-world benchmark, MIGU outperforms all evaluated baselines, while ablations support the benefit of explicit multimodal uncertainty modeling. Project website: multimodal-instruction.github.io

Problem

• Multimodal Instruction
• Uncertainty
• Manipulation Planning

Innovation

• Multimodal Instruction Grounding
• Uncertainty Modeling
• Bayes-inspired Fusion
• Vision-Language Model
• Manipulation Planning