🤖 AI Summary
This work addresses the low sample efficiency of traditional model-free reinforcement learning methods—such as Proximal Policy Optimization (PPO)—which rely on high-variance advantage estimates. The authors propose Analytic Policy Gradients (APG), a method that leverages differentiable environment dynamics to compute exact, end-to-end gradients of policy returns with respect to policy parameters. To mitigate gradient degradation in long-horizon tasks, APG incorporates a segment-wise backpropagation mechanism and combines Monte Carlo estimation with critic-guided bootstrapping for effective gradient guidance. Evaluated on four continuous control benchmarks under identical network architectures and training protocols, APG consistently outperforms PPO, demonstrating substantially higher sample efficiency and faster convergence.
📝 Abstract
Model-free reinforcement learning algorithms such as Proximal Policy Optimization (PPO) treat the environment as a black box, estimating policy gradients from sampled rewards; this process demands millions of interactions and relies on high-variance advantage estimates. When environment dynamics are differentiable, the return is an end-to-end differentiable function of the policy parameters, enabling exact gradient computation via backpropagation through simulation. We term this approach Analytic Policy Gradients (APG) and evaluate it against PPO on four continuous control tasks of increasing dynamical complexity: a one-dimensional point-mass target-reaching task, a 2D point-mass navigation task with obstacle avoidance, a 2D rigid-body T-block pushing task, and a 7-DOF Franka FR3 end-effector reaching task. Both algorithms share identical model architectures, observation normalization, and optimizer settings. To decouple sample efficiency from compute efficiency, we design a multi-axis evaluation protocol that records performance against environment steps and gradient steps. We report a segmented backpropagation scheme with MC and critic-based bootstrap modes that mitigates gradient degradation on long-horizon tasks, and present ablations over segment length and bootstrap strategy.