Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

2026-07-17

AI Summary

This work addresses the challenges of real-time optimal control in high-dimensional dynamical systems, where low sample efficiency, the curse of dimensionality in exploration, and gradient instability hinder performance. The authors propose the PEARL framework, which uniquely integrates the adjoint method with a neural network-based reward function. By exploiting the differentiability of system dynamics, PEARL employs an actor-adjoint algorithm that combines automatic differentiation with adjoint sensitivity analysis to efficiently compute policy gradients over short horizons. This approach enables physics-informed policy learning, significantly enhancing sample efficiency and generalization while operating directly in high-dimensional state-action spaces. Evaluated on unsteady flow navigation tasks, PEARL outperforms existing reinforcement learning methods and scales to high-dimensional control problems without requiring dimensionality reduction or multi-agent architectures.

Abstract

Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interaction with the environment to synthesize optimal control strategies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces. In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentibility of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities. Through two challenging parametric navigation problems in unsteady flows, we show that PEARL (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms, (ii) is sample efficient, thanks to the physics-guided policy learning, (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems, and (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.

Problem

• reinforcement learning
• optimal control
• dynamical systems
• sample efficiency
• high-dimensional control

Innovation

• Physics-enhanced reinforcement learning
• Differentiable dynamics
• Actor-adjoint algorithm
• Adjoint-based sensitivity
• Sample efficiency