🤖 AI Summary
This work addresses the low sample efficiency and inconsistent actions often observed in reinforcement learning for robotic control, which stem from neglecting known physical dynamics. To this end, the authors propose PIPER, a novel framework that seamlessly integrates physical priors into policy learning by incorporating a differentiable Lagrangian dynamics residual—computed via a standard simulator—as a soft regularization term directly into the policy objective. Crucially, this approach requires no modifications to the underlying simulator or reinforcement learning algorithm. By softly enforcing analytical physical constraints during policy updates, PIPER achieves a tight coupling between physical consistency and learning, significantly improving sample efficiency, training stability, and control accuracy. Empirical results across multiple robotic tasks demonstrate that policies trained with PIPER exhibit superior physical plausibility and overall performance compared to baseline methods.
📝 Abstract
Reinforcement learning (RL) has achieved strong performance in robotic control; however, state-of-the-art policy learning methods, such as actor-critic methods, still suffer from high sample complexity and often produce physically inconsistent actions. This limitation stems from neural policies implicitly rediscovering complex physics from data alone, despite accurate dynamics models being readily available in simulators. In this paper, we introduce a novel physics-informed RL framework, called PIPER, that seamlessly integrates physical constraints directly into neural policy optimization with analytical soft physics constraints. At the core of our method is the integration of a differentiable Lagrangian residual as a regularization term within the actor's objective. This residual, extracted from a robot's simulator description, subtly biases policy updates towards dynamically consistent solutions. Crucially, this physics integration is realized through an additional loss term during policy optimization, requiring no alterations to existing simulators or core RL algorithms. Extensive experiments demonstrate that our method significantly improves learning efficiency, stability, and control accuracy, establishing a new paradigm for efficient and physically consistent robotic control.