Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation

2024-12-16

arXiv.org

AI Summary

This work addresses the low sample efficiency and training instability of reinforcement learning (RL) in soft-rigid coupled multiphysics simulation, primarily caused by slow simulation speed and non-differentiable dynamics. We propose SAPO—the first maximum-entropy, first-order model-based RL algorithm tailored for differentiable multiphysics simulation. Methodologically, SAPO integrates the Soft Actor-Critic framework with analytical first-order gradient backpropagation through the simulator and leverages GPU-accelerated differentiable simulation. We further introduce Rewarped, an open-source platform enabling unified, high-performance simulation of rigid bodies, articulated bodies, and deformable objects. Experiments demonstrate that SAPO significantly outperforms state-of-the-art methods across diverse soft-rigid manipulation and locomotion tasks, improving sample efficiency by one to two orders of magnitude. Notably, SAPO achieves, for the first time, end-to-end RL policy learning for high-fidelity soft-body interaction tasks.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advances in GPU-based parallel simulation have enabled practitioners to collect large amounts of data and train complex control policies using deep reinforcement learning (RL), on commodity GPUs. However, such successes for RL in robotics have been limited to tasks sufficiently simulated by fast rigid-body dynamics. Simulation techniques for soft bodies are comparatively several orders of magnitude slower, thereby limiting the use of RL due to sample complexity requirements. To address this challenge, this paper presents both a novel RL algorithm and a simulation platform to enable scaling RL on tasks involving rigid bodies and deformables. We introduce Soft Analytic Policy Optimization (SAPO), a maximum entropy first-order model-based actor-critic RL algorithm, which uses first-order analytic gradients from differentiable simulation to train a stochastic actor to maximize expected return and entropy. Alongside our approach, we develop Rewarped, a parallel differentiable multiphysics simulation platform that supports simulating various materials beyond rigid bodies. We re-implement challenging manipulation and locomotion tasks in Rewarped, and show that SAPO outperforms baselines over a range of tasks that involve interaction between rigid bodies, articulations, and deformables. Additional details at https://rewarped.github.io/.

Problem

• Enables RL for tasks involving rigid and deformable bodies.
• Introduces SAPO for efficient policy optimization using differentiable simulation.
• Develops Rewarped, a platform for parallel differentiable multiphysics simulation.

Innovation

• SAPO: maximum entropy actor-critic RL algorithm
• Rewarped: parallel differentiable multiphysics simulation platform
• First-order analytic gradients for differentiable simulation