Technology Category
Natural Language Processing: Language Grounding & Multi-modal NLPComputer Vision: Language and VisionMachine Learning: Large Multimodal Models (LMMs)
Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions
🤖 AI Summary
This work addresses the problem of language-guided, precise localization of object affordances in 3D embodied environments—particularly under partial occlusion, multi-view observation, and arbitrary object rotations that induce incomplete sensory input. We formally introduce a novel task: language–vision–interaction joint-driven 3D affordance grounding. To support this, we present AGPIL, the first multimodal dataset covering full-view, occluded, and multi-rotation scenarios with fine-grained affordance annotations. Methodologically, we propose LMAffordance3D, a language-guided multimodal 3D affordance grounding network integrating vision-language models (VLMs), point-cloud encoders, cross-modal attention, and multi-view geometric alignment to jointly embed and spatially ground 2D images, 3D point clouds, and natural-language instructions. Evaluated on AGPIL, LMAffordance3D outperforms all baselines by +12.6 mAP and demonstrates strong generalization to unseen objects, viewpoints, and instruction combinations.
Technology Category
Application Category
📝 Abstract
Grounding 3D object affordance is a task that locates objects in 3D space where they can be manipulated, which links perception and action for embodied intelligence. For example, for an intelligent robot, it is necessary to accurately ground the affordance of an object and grasp it according to human instructions. In this paper, we introduce a novel task that grounds 3D object affordance based on language instructions, visual observations and interactions, which is inspired by cognitive science. We collect an Affordance Grounding dataset with Points, Images and Language instructions (AGPIL) to support the proposed task. In the 3D physical world, due to observation orientation, object rotation, or spatial occlusion, we can only get a partial observation of the object. So this dataset includes affordance estimations of objects from full-view, partial-view, and rotation-view perspectives. To accomplish this task, we propose LMAffordance3D, the first multi-modal, language-guided 3D affordance grounding network, which applies a vision-language model to fuse 2D and 3D spatial features with semantic features. Comprehensive experiments on AGPIL demonstrate the effectiveness and superiority of our method on this task, even in unseen experimental settings. Our project is available at https://sites.google.com/view/lmaffordance3d.
Problem
Research questions and friction points this paper is trying to address.
Ground 3D object affordance using language, vision, and interactions
Address partial observations in 3D space due to occlusion or rotation
Develop multi-modal network for language-guided 3D affordance grounding
Innovation
Methods, ideas, or system contributions that make the work stand out.
Multi-modal vision-language model fusion
Language-guided 3D affordance grounding
Partial-view and rotation-view affordance estimation
🔎 Similar Papers
No similar papers found.
