6D Object Pose Tracking in Internet Videos for Robotic Manipulation

2025-03-13

AI Summary

This work addresses the challenging task of extracting 6D pose trajectories of manipulated objects from unstructured internet-sourced instructional videos—characterized by unconstrained camera motion, unknown object CAD models, and subtle object dynamics causing temporal inconsistency. We propose the first RGB-only, end-to-end framework requiring no object priors: it jointly performs cross-modal CAD retrieval, image-level 6D pose alignment, and scene-scale anchoring for initial pose estimation; then refines trajectories via video-based smoothing and robot configuration-space optimization for action retargeting. Our method achieves significant improvements over state-of-the-art on YCB-V, HOPE-Video, and a newly curated instructional video dataset. It successfully drives a 7-DOF robotic arm to replicate demonstrated manipulations in both simulation and real-world settings. Furthermore, we demonstrate strong generalization to first-person videos from EPIC-KITCHENS, validating its potential for embodied AI applications.

Technology Category

Intelligent Robots: ManipulationComputer Vision: Image and Video RetrievalHumans and AI: Human-Aware Planning and Behavior Prediction

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingResponsible Web: Machine-in-the-loop, human agency and autonomyUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We seek to extract a temporally consistent 6D pose trajectory of a manipulated object from an Internet instructional video. This is a challenging set-up for current 6D pose estimation methods due to uncontrolled capturing conditions, subtle but dynamic object motions, and the fact that the exact mesh of the manipulated object is not known. To address these challenges, we present the following contributions. First, we develop a new method that estimates the 6D pose of any object in the input image without prior knowledge of the object itself. The method proceeds by (i) retrieving a CAD model similar to the depicted object from a large-scale model database, (ii) 6D aligning the retrieved CAD model with the input image, and (iii) grounding the absolute scale of the object with respect to the scene. Second, we extract smooth 6D object trajectories from Internet videos by carefully tracking the detected objects across video frames. The extracted object trajectories are then retargeted via trajectory optimization into the configuration space of a robotic manipulator. Third, we thoroughly evaluate and ablate our 6D pose estimation method on YCB-V and HOPE-Video datasets as well as a new dataset of instructional videos manually annotated with approximate 6D object trajectories. We demonstrate significant improvements over existing state-of-the-art RGB 6D pose estimation methods. Finally, we show that the 6D object motion estimated from Internet videos can be transferred to a 7-axis robotic manipulator both in a virtual simulator as well as in a real world set-up. We also successfully apply our method to egocentric videos taken from the EPIC-KITCHENS dataset, demonstrating potential for Embodied AI applications.

Problem

• Extract 6D pose trajectories from Internet videos.
• Estimate 6D pose without prior object knowledge.
• Transfer 6D object motion to robotic manipulators.

Innovation

• Estimates 6D pose without prior object knowledge
• Extracts smooth 6D trajectories from videos
• Transfers motion to robotic manipulators effectively