TacSushi: Tactile-Grounded World-Action Modeling for Dexterous Sushi Manipulation

2026-09-16

AI Summary

研究提出TacSushi，一种基于触觉的策略模型，通过融合当前视觉、语言和触觉信息学习未来操作后果，以解决精细食物操作中的变形、遮挡和不确定接触问题。

Abstract

Dexterous food manipulation requires control under deformation, occlusion, and uncertain contact. We present TacSushi, a tactile-grounded, Cosmos3-based world-action policy that learns from recorded future consequences while acting on current observations. The backbone encodes current RGB, language, and hand state, and feature-wise gated fusion incorporates fingertip tactile features into the action representation. During training, a decoder conditioned on demonstrated action chunks predicts logged future visual observations, task progress, relative contact risk, and tactile summaries; this decoder is removed at deployment. Failed trials provide consequence supervision, but their actions are excluded from imitation. We train TacSushi on 340 successful and 50 failed real-robot trials and compare six methods in 600 separate rollouts across three in-distribution tasks and two out-of-distribution ingredient variants. To assess food quality beyond a single geometric threshold, we score terminal outcomes using an anchored visual-quality protocol that equally weights five human ratings and three vision-language-model ratings per rollout. Full TacSushi achieves 68.3% average in-distribution success and 37.5% out-of-distribution success, compared with 36.7%/10.0% without future-consequence supervision and 25.0%/17.5% with direct tactile concatenation in place of gated fusion. These comparisons support complementary benefits of feature-wise gated tactile fusion and training-only predictive supervision.

Problem

• Dexterous Food Manipulation
• Deformation
• Occlusion
• Uncertain Contact

Innovation

• Tactile-Grounded
• Feature-wise Gated Fusion
• Future-Consequence Supervision
• Dexterous Food Manipulation