Flow Policy Gradients for Robot Control

2026-02-02

AI Summary

This work addresses the limitations of conventional likelihood-based policy gradient methods, which rely on simplistic action distributions—such as Gaussian—and struggle to capture the complexity of real-world robotic control policies. To overcome this, the authors propose an enhanced flow matching policy gradient framework that bypasses explicit likelihood computation. By introducing a novel flow matching objective, the approach enables effective exploration during training from scratch and substantially improves fine-tuning robustness and sim-to-real transferability. The method demonstrates strong performance across diverse tasks, including legged locomotion, humanoid motion tracking, and manipulation. Notably, it achieves robust simulation-to-reality transfer on two physical humanoid robots, highlighting its practical applicability and generalization capabilities in real-world settings.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Likelihood-based policy gradient methods are the dominant approach for training robot control policies from rewards. These methods rely on differentiable action likelihoods, which constrain policy outputs to simple distributions like Gaussians. In this work, we show how flow matching policy gradients -- a recent framework that bypasses likelihood computation -- can be made effective for training and fine-tuning more expressive policies in challenging robot control settings. We introduce an improved objective that enables success in legged locomotion, humanoid motion tracking, and manipulation tasks, as well as robust sim-to-real transfer on two humanoid robots. We then present ablations and analysis on training dynamics. Results show how policies can exploit the flow representation for exploration when training from scratch, as well as improved fine-tuning robustness over baselines.

Problem

• policy gradient
• robot control
• expressive policies
• likelihood-based methods
• flow matching

Innovation

• flow matching
• policy gradient
• robot control
• sim-to-real transfer
• expressive policies