Differentiable Reinforcement Learning for Path Tracking by an Agile Fish-Like Robot

2026-07-17

AI Summary

This work addresses the significant challenges in motion control and path tracking for fish-like robots, which arise from the complex fluid–structure interaction modeling and nonlinear underactuated dynamics. The lack of an efficient yet accurate simulation environment has further hindered the application of reinforcement learning in this domain. To overcome these limitations, the authors develop a highly efficient, differentiable simulation platform for fish-like robots and, for the first time, integrate differentiable reinforcement learning with curriculum learning. By employing backpropagation through time (BPTT), they optimize the time-varying gains of a PID controller. The learned policy, trained entirely in simulation, is directly transferred to a physical robot without fine-tuning, achieving high-fidelity sim-to-real transfer and substantially improving both path-tracking accuracy and robustness.

Abstract

Fish-like swimming has inspired the design of several dozens if not hundreds of bioinspired robots in the last few decades. But the control and motion planning of such robots has been challenging due to the poorly modeled fluid-structure interaction and the nonlinear underactuated dynamics of such robots. While reinforcement learning has allowed significant advances in the context of ground and aerial robots, the lack of a suitable simulation environment with appropriate computational speed and accuracy have prevented similar progress for fish-like robots. We address this two-fold problem by developing a simulation platform that approximates the motion of our fish-like robot with computational efficiency. Then the motion control and path tracking by the robot is performed using PID control where the (variable) gains are learned using back propagation through time and training on a curriculum. The policy learned in the simulation is then applied on the physical platform, demonstrating an excellent match.

Problem

• fish-like robot
• path tracking
• fluid-structure interaction
• underactuated dynamics
• simulation environment

Innovation

• differentiable reinforcement learning
• fish-like robot
• simulation-to-reality transfer
• backpropagation through time
• curriculum learning