KineSoft: Learning Proprioceptive Manipulation Policies with Soft Robot Hands

2025-03-03

AI Summary

Addressing the challenges of difficult demonstration acquisition and insufficient state representation in dexterous manipulation with soft robotic hands, this paper proposes a proprioception-aware physical teleoperation learning framework. Methodologically, we introduce: (1) a novel integrated flexible strain-sensing array enabling occlusion-free, high-precision real-time estimation of the soft hand’s 3D shape; (2) a shape-feedback-driven diffusion-based policy imitation learning method that directly maps human kinesthetic teaching actions into proprioceptive control space; and (3) a shape-conditioned low-level controller leveraging the inherent compliance of soft actuators to facilitate skill acquisition—not merely compensate for deformation. Experimental results demonstrate a 42% reduction in shape estimation error, a 3.1× improvement in trajectory tracking accuracy, and an average 58% increase in multi-task success rate compared to baseline approaches.

Technology Category

• Intelligent Robots: Manipulation
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Underactuated soft robot hands offer inherent safety and adaptability advantages over rigid systems, but developing dexterous manipulation skills remains challenging. While imitation learning shows promise for complex manipulation tasks, traditional approaches struggle with soft systems due to demonstration collection challenges and ineffective state representations. We present KineSoft, a framework enabling direct kinesthetic teaching of soft robotic hands by leveraging their natural compliance as a skill teaching advantage rather than only as a control challenge. KineSoft makes two key contributions: (1) an internal strain sensing array providing occlusion-free proprioceptive shape estimation, and (2) a shape-based imitation learning framework that uses proprioceptive feedback with a low-level shape-conditioned controller to ground diffusion-based policies. This enables human demonstrators to physically guide the robot while the system learns to associate proprioceptive patterns with successful manipulation strategies. We validate KineSoft through physical experiments, demonstrating superior shape estimation accuracy compared to baseline methods, precise shape-trajectory tracking, and higher task success rates compared to baseline imitation learning approaches.

Problem

• Develop dexterous manipulation skills for soft robot hands.
• Overcome demonstration collection challenges in soft systems.
• Improve state representation and control in soft robotics.

Innovation

• Direct kinesthetic teaching for soft robots
• Internal strain sensing for shape estimation
• Shape-based imitation learning with proprioceptive feedback