SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments

2026-07-22

AI Summary

This work addresses the challenge of multi-arm grasping in complex scenes that requires joint semantic understanding and 3D spatial reasoning. The authors propose SeededGrasp, a framework that leverages a vision-language model to generate semantically guided grasp seed points, which serve as conditional inputs to a lightweight grasp generation network, thereby decoupling high-level semantics from low-level geometric execution. Notably, this approach enables language-guided multi-agent grasping without end-to-end training, significantly improving data efficiency and generalization. The study also introduces the first large-scale multi-agent tabletop grasping dataset, comprising 2.5 million samples, and demonstrates success rates of 72% in simulation and 78% in real-world experiments—substantially outperforming existing baselines.

Abstract

Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments. See our project site for data and code: https://uoft-isl.github.io/seeded-grasp/

Problem

• language-guided grasping
• complex scenes
• multi-embodiment
• vision-language models
• 3D spatial reasoning

Innovation

• SeededGrasp
• vision-language models
• multi-embodiment grasping
• seed point conditioning
• data-efficient robotics