When Does Touch Matter? Charting the Vision-Interaction Gap in Cluttered Dexterous Grasping

2026-09-20

AI Summary

研究解决了在杂乱环境中灵巧抓取时触觉测量何时重要的问题，通过结合视觉、力矩估计和触觉传感器的方法进行实验，表明触觉反馈与视觉信息互补并提高了抓取成功率。

Abstract

Dexterous grasping in clutter poses a basic sensing question: when do tactile measurements and external wrench estimates improve on visual geometry? Occlusion and contact can obscure grasp quality, motivating a controlled evaluation of these interaction signals. We present a controlled real-world study over five tabletop scene conditions on a dexterous system that combines vision, per-finger and wrist wrench estimates, and distributed fingertip taxels. With demonstrations, visual observations, action space, and compliant control fixed, we compare vision-only, wrench, taxel, and combined policies plus representation and fusion baselines. The combined policy succeeds in 24/25 trials versus 14/25 for vision only, and 15/15 versus 6/15 across the three confined conditions. Ablations show that wrench and taxel feedback are complementary. Behavioral comparisons show that interaction feedback enables earlier rejection of inadequate contacts, regrasping before lift, and more stable grasps. To our knowledge, this is the first real-world study to combine and separately evaluate these interaction modalities for target-oriented dexterous grasping in clutter. These results chart a widening vision-interaction gap and position cluttered dexterous grasping as a benchmark for determining when the learned policy needs interaction sensing. Project website: https://interaction-dex-grasp.github.io/

Problem

• dexterous grasping
• cluttered environment
• vision-interaction gap
• tactile measurements
• wrench estimates

Innovation

• dexterous grasping
• wrench estimates
• taxel feedback
• interaction sensing