DLO-Splatting: Tracking Deformable Linear Objects Using 3D Gaussian Splatting

2025-05-13

AI Summary

This work addresses the challenge of achieving high-accuracy, real-time 3D shape estimation for deformable linear objects (DLOs) under severe non-rigid deformations—such as knotting—where conventional methods suffer from ambiguity and instability. We propose a prediction-update filtering framework that fuses multi-view RGB images with robotic gripper state information. Our method innovatively employs differentiable 3D Gaussian splatting rendering, integrated with position-based dynamics modeling, multi-view geometric constraint optimization, and shape smoothing/rigidity-damping regularization. Compared to vision-only approaches, our framework significantly improves robustness and real-time performance in complex deformation scenarios, achieving centimeter-level reconstruction accuracy during knotting tasks. This enables reliable, low-latency 3D perception essential for dynamic DLO manipulation.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This work presents DLO-Splatting, an algorithm for estimating the 3D shape of Deformable Linear Objects (DLOs) from multi-view RGB images and gripper state information through prediction-update filtering. The DLO-Splatting algorithm uses a position-based dynamics model with shape smoothness and rigidity dampening corrections to predict the object shape. Optimization with a 3D Gaussian Splatting-based rendering loss iteratively renders and refines the prediction to align it with the visual observations in the update step. Initial experiments demonstrate promising results in a knot tying scenario, which is challenging for existing vision-only methods.

Problem

• Estimating 3D shape of deformable linear objects
• Tracking DLOs using multi-view RGB images
• Improving accuracy with dynamics and rendering optimization

Innovation

• Uses 3D Gaussian Splatting for deformable object tracking
• Combines position-based dynamics with visual optimization
• Integrates gripper state and multi-view RGB inputs