Kitchen Robotic Manipulation utilizing Foundation Models

2026-08-04

AI Summary

This work addresses the challenges of robustness and generalization in manipulating tableware within cluttered and occluded home kitchen environments. The authors propose a modular perception pipeline that integrates open-vocabulary object detection (LLMDet), multi-view segmentation (SAMv2), instance-level 3D reconstruction, and 2D–3D feature fusion to achieve high-precision 6D pose estimation and grasp planning. The architecture flexibly incorporates various vision and geometry foundation models—such as DINOv2 and GeoTransformer—and can be deployed in real-world kitchens without requiring scene-specific fine-tuning. Evaluated on a benchmark comprising 20 complex kitchen scenes, the system achieves an ADI accuracy of 89.12% and successfully performs physical robotic tasks including dishwasher loading and cup stacking.

Abstract

Deploying robots in everyday human environments requires perception systems that are both robust and adaptable to diverse, dynamic conditions. In this work, we present a modular perception pipeline for household manipulation tasks, with a focus on dishware handling in kitchen environments. The pipeline integrates open-vocabulary object detection, multi-view segmentation, instance-aware 3D reconstruction, and a 2D-3D feature fusion strategy for 6D pose estimation and grasp planning. Its modular design enables systematic substitution of multiple visual and geometric foundation models, allowing us to identify the best-performing configuration through extensive evaluation on a custom kitchen dataset. The best-performing configuration (LLMDet + SAMv2 + DINOv2 + GeoTransformer) achieves an ADI of 89.12\% on the 20-scene kitchen benchmark with cluttered and occluded conditions. Furthermore, real-world demonstrations confirm that the best configuration can be deployed on physical robots without environment-specific retraining, successfully executing tasks such as sink-to-dishwasher transfer and cup stacking. It validates the adaptability and scalability of the pipeline and highlights its potential as a practical framework for household robotic systems. Our code and supplementary materials are available at https://raivlab.github.io/FM_kitchen .

Problem

• Kitchen Robotic Manipulation
• Foundation Models
• 6D Pose Estimation
• Open-Vocabulary Object Detection
• Household Robotics

Innovation

• foundation models
• modular perception pipeline
• 6D pose estimation
• open-vocabulary object detection
• robotic manipulation