SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation

2026-08-04

AI Summary

This work addresses the challenge of efficiently curating beneficial data from vast collections of first-person human videos for post-training vision-language-action (VLA) models in robotic dexterous manipulation. To this end, the authors propose a similarity-based data mining framework that adapts ideas from recommendation systems to cross-embodiment dexterous manipulation, introducing a three-stage recall–ranking–reranking pipeline to match each robot demonstration with a task-relevant subset of human videos. The approach constructs morphology-agnostic action representations, enabling effective utilization of human video data without modifying the underlying VLA architecture. Using only 1.49 million carefully selected samples—less than 5% of the original pool—the method boosts task success rates from 47.7% to 61.1%, substantially outperforming random sampling baselines.

Abstract

Recent years have witnessed an explosive trend of scaling ego-centric human videos for robot manipulation, yet it remains unclear which data actually benefits dexterous manipulation. We present SiMDex, a similarity-based data mining framework that casts human data selection for VLA post-training in dexterous manipulation as a recommendation problem. For each robot demonstration, SiMDex employs a three-layer recall-ranking-re-ranking pipeline to extract task-relevant subsets from a pool of ~32M egocentric human samples, operating in a morphology-agnostic action space that requires no changes to VLA architecture or training. Against a strong baseline trained with an equal amount of randomly sampled human data, SiMDex uses only ~1.49M mined samples (<5% of the pool) yet improves the overall success rate from 47.7% to 61.1%, showing that selective curation outperforms indiscriminate data mixing.

Problem

• dexterous manipulation
• egocentric videos
• data selection
• robot learning
• cross-embodiment

Innovation

• similarity-based data mining
• egocentric video
• dexterous manipulation
• cross-embodiment
• VLA post-training