RTFF: Random-to-Target Fabric Flattening Policy using Dual-Arm Manipulator

2025-10-01

AI Summary

Deformable cloth flattening faces challenges including complex wrinkle geometry, high degrees of freedom, and severe visual occlusion. Method: This paper proposes a dual-arm cooperative control framework integrating imitation learning (IL) and visual servoing (VS). It introduces template mesh modeling for cross-step vertex correspondence and wrinkle geometry perception; designs a randomized-to-target flattening strategy enabling dynamic IL–VS mode switching during execution; and employs a Transformer-based Multi-Agent Coordination Transformer (MACT) policy with mesh-action chunking to exploit cloth topology for policy learning and fine-grained manipulation. Results: Evaluated on a real dual-arm robotic platform, the method achieves zero-shot adaptation to multi-shape alignment without task-specific training, delivers high flattening accuracy, and demonstrates strong generalization across diverse fabric materials, scales, and initial wrinkle configurations.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Diffusion Models for Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Robotic fabric manipulation in garment production for sewing, cutting, and ironing requires reliable flattening and alignment, yet remains challenging due to fabric deformability, effectively infinite degrees of freedom, and frequent occlusions from wrinkles, folds, and the manipulator's End-Effector (EE) and arm. To address these issues, this paper proposes the first Random-to-Target Fabric Flattening (RTFF) policy, which aligns a random wrinkled fabric state to an arbitrary wrinkle-free target state. The proposed policy adopts a hybrid Imitation Learning-Visual Servoing (IL-VS) framework, where IL learns with explicit fabric models for coarse alignment of the wrinkled fabric toward a wrinkle-free state near the target, and VS ensures fine alignment to the target. Central to this framework is a template-based mesh that offers precise target state representation, wrinkle-aware geometry prediction, and consistent vertex correspondence across RTFF manipulation steps, enabling robust manipulation and seamless IL-VS switching. Leveraging the power of mesh, a novel IL solution for RTFF-Mesh Action Chunking Transformer (MACT)-is then proposed by conditioning the mesh information into a Transformer-based policy. The RTFF policy is validated on a real dual-arm tele-operation system, showing zero-shot alignment to different targets, high accuracy, and strong generalization across fabrics and scales. Project website: https://kaitang98.github.io/RTFF_Policy/

Problem

• Aligning randomly wrinkled fabrics to wrinkle-free target states
• Addressing fabric deformability and infinite degrees of freedom
• Solving fabric manipulation challenges for garment production tasks

Innovation

• Hybrid Imitation Learning-Visual Servoing framework
• Template-based mesh for wrinkle-aware geometry prediction
• Transformer-based policy with mesh action chunking