PneuTac: Tactile Manipulation with Soft Pneumatic Robots via Unified MPM-Gaussian Splatting Simulation

2026-09-29

AI Summary

This study addresses the policy training bottleneck in soft robotic tactile manipulation caused by the absence of efficient, unified simulation. We propose a rigid-soft coupled simulation framework that integrates the Material Point Method (MPM) with 3D Gaussian Splatting (3DGS). This approach pioneers the combination of MPM-based physical dynamics with high-fidelity 3DGS rendering and incorporates vision-based real-to-sim modeling to construct a unified simulation environment for tactile manipulation, significantly narrowing the sim-to-real gap. Experiments across three contact-intensive tasks demonstrate that simulation-augmented policies trained within this framework substantially outperform existing baseline methods, fully validating its effectiveness and practical utility.

Abstract

Soft robots and tactile sensors have demonstrated great potential in delicate manipulation tasks. Soft pneumatic robots enable safe contact through compliance, and vision-based tactile sensors offer high-resolution touch perception. However, learning tactile manipulation with compliant robots has been challenging, bottlenecked by the lack of efficient simulation. Existing simulators typically model them in isolation, and exhibit large calibration gaps that are difficult to overcome efficiently. We present PneuTac, a unified framework for tactile-feedback manipulation with soft pneumatic robots. We leverage the material point method (MPM) for modelling the dynamics of the soft robot and the deformable tactile membrane, and 3D Gaussian splatting (3DGS) for rendering. Real-to-sim modelling is done with a simple vision-based method, to then train action and perception networks for efficient simulation with surrogate models. We use the framework to drive a tactile-guided pipeline to collect demonstrations in simulation. Through experiments on a custom-designed pneumatic soft finger with a tactile sensing tip, together with additional cross-device evaluations, we show that PneuTac is capable of accurately modelling soft robots with tactile sensors, and that policies trained with simulation-augmented demonstrations outperform baselines trained on the same real data on three real-world contact-rich compliant manipulation tasks, making it a practical framework for tactile manipulation on compliant hardware.

Problem

• soft pneumatic robots
• tactile manipulation
• simulation
• calibration gap
• vision-based tactile sensors

Innovation

• Soft Pneumatic Robots
• Material Point Method (MPM)
• 3D Gaussian Splatting (3DGS)
• Tactile Manipulation
• Sim-to-Real