GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

2026-06-05

AI Summary

Existing flow-based generative policies in on-policy reinforcement learning struggle to compute exact action execution probabilities, often introducing bias or incurring substantial computational overhead. This work proposes GenPO++, a novel framework that, for the first time, enables exact and Jacobian-free likelihood ratio estimation without expanding the action space dimensionality. GenPO++ integrates invertible generative policies, high-order invertible ODE solvers, and a history-based auxiliary memory mechanism, allowing direct computation of Jacobian-free likelihood ratios through fixed solver coefficients. Empirical evaluations demonstrate that GenPO++ matches or surpasses state-of-the-art methods across large-scale simulated control, fine-tuning, and real-world robotic tasks, while significantly enhancing training stability and computational efficiency.

Abstract

Generative policies provide expressive and multimodal action distributions, making them attractive for reinforcement learning (RL) in complex continuous-control tasks. Among them, flow-based policies are especially appealing because they generate actions through deterministic transport maps. However, applying such generative policies to likelihood-based on-policy learning remains limited by the difficulty of evaluating the probability of executed actions. Existing flow RL methods either replace the true action-density ratio with approximate surrogates, which can introduce biased updates, or recover exact likelihoods through dummy-action augmentation, which enlarges the policy space and increases computation. In this work, we propose GenPO++, a reversible generative policy optimization framework that uses history states as auxiliary memory in a high-order reversible ODE solver, yielding exact inversion without changing the original action dimension. The resulting generative policy map has a log-determinant determined only by fixed solver coefficients, enabling exact and Jacobian-free likelihood-ratio computation. This design preserves the expressiveness of generative flow policies while avoiding both action ratio bias and dummy-action overhead. We evaluate GenPO++ on large-scale simulated control, fine-tuning, and real-world robotic manipulation tasks, where it achieves competitive or superior performance over state-of-the-art on-policy RL methods, while improving training stability and computational efficiency.

Problem

• generative policy
• likelihood ratio
• flow-based model
• on-policy reinforcement learning
• action probability evaluation

Innovation

• Generative Policy Optimization
• Jacobian-free Likelihood Ratio
• Reversible ODE Solver
• Flow-based Policy
• On-policy Reinforcement Learning