Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation

2026-06-04

AI Summary

This work addresses the challenges of visual perception in fabric manipulation, particularly those arising from continuous deformations, self-occlusions, and the scarcity of real-world annotated data. To overcome these issues, the authors propose a vision-based perception framework capable of generalizing to real fabrics without fine-tuning. They develop a Blender-based synthetic data generation pipeline that combines manually annotated rendered images with real data to train a detector integrating permutation-invariant CNN-based keypoint detection and YOLOv8–OpenCV wrinkle analysis. The resulting perception system is integrated into a dual-arm robotic platform to enable garment stretching and ironing tasks. Experimental results demonstrate that the keypoint model achieves an average positional error of 1.76 pixels and significantly outperforms existing baselines under high occlusion conditions.

Abstract

Robotic manipulation of textiles remains challenging because continuous deformation and self-occlusions hinder the robust visual perception required to estimate the cloth's state. To address the lack of annotated real-world data, we developed a Blender-based synthetic pipeline exporting auto-annotated keypoints, and combined manually labeled renders with real-world data to train a wrinkle detector. We present a perception framework integrating a CNN for permutation-invariant keypoint detection and a YOLOv8-OpenCV pipeline to extract grasping points from structural wrinkles. A proposed bimanual algorithm uses this system to stretch fully folded garments via wrinkles, transitioning to keypoint-based ironing once corners emerge. The keypoint model achieves a Mean Position Error (MPE) of 1.7615 pixels. The perception system transfers to physical fabrics without fine-tuning, outperforming baselines that fail in high-occlusion states or yield false positives on severe folds.

Problem

• cloth manipulation
• visual perception
• self-occlusion
• deformation
• state estimation

Innovation

• synthetic data generation
• wrinkle detection
• keypoint detection
• bimanual cloth manipulation
• permutation-invariant CNN