Sparse Meets Dense: Correspondence Guided Robotic Manipulation with Rigid-Deformable Interactions

2026-08-02

AI Summary

This work addresses the challenges of robotic manipulation involving interactions between rigid and deformable objects—such as dressing or hanging garments—where multi-point contacts and complex deformations hinder reliable control. To tackle this, the authors propose a hybrid sparse-dense correspondence representation that integrates task- and interaction-aware sparse keypoints, generated from global structure and refined through local contact constraints, with dense surface correspondences of the deformable object. This formulation balances task specificity with generalization capability, enabling effective policy transfer to novel tasks, deformations, and scenes from a single demonstration. Experimental results demonstrate that the proposed approach significantly enhances the adaptability, robustness, and practicality of robotic systems across a variety of rigid-deformable interaction scenarios.

Abstract

Manipulation involving rigid-deformable interactions, such as hanging clothes or dressing humans, is common in daily life, making it essential for household robots. Compared to single-object manipulation or interactions between rigid bodies, these tasks are particularly challenging due to the rich multi-point contacts and the complex dynamics of the deformable bodies during interaction. Therefore, object-centric representations such as 6D poses or structural points without task-specific information become insufficient for these interactions. In this work, we propose a hybrid correspondence-based representation tailored for rigid-deformable interactions. First, to capture intricate interaction information, we introduce structure-, task-, and interaction-aware sparse keypoints. The keypoints are generated based on the global structures of both rigid and deformable objects, and filtered by their local interaction contacts. However, tracking these sparse keypoints through the interaction remains difficult due to the high-dimensional dynamics of deformable objects. Therefore, we further construct dense correspondences on the deformable objects for accurate keypoint tracking throughout the manipulation. This hybrid design combines the advantages of both representations: sparse keypoints encode rich, task-specific information for fine-grained manipulation, while dense correspondences ensure efficient tracking and generalization to novel deformations, shapes, and scenarios. Together, they enable one-shot transfer to new tasks with minimal demonstrations. Extensive experiments demonstrate the effectiveness and broad applicability of our method.

Problem

• rigid-deformable interaction
• robotic manipulation
• deformable objects
• correspondence representation
• multi-point contact

Innovation

• correspondence-based representation
• rigid-deformable interaction
• sparse keypoints
• dense correspondence
• one-shot transfer