RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process

2026-09-26

AI Summary

This study addresses the challenge of fine-tuning generative robot policies via reinforcement learning, where intractable likelihood estimation hinders overcoming the distribution shift inherent in imitation learning. To this end, we propose RoboFFT, a novel framework that pioneers the construction of the PPO surrogate ratio through forward noise perturbation. By integrating a weighted score matching loss, our approach enables efficient online fine-tuning of diffusion and flow matching policies without requiring additional expert demonstrations. Experimental results demonstrate that RoboFFT significantly improves both performance and training stability on long-horizon planning and sparse-reward benchmarks. Furthermore, the effectiveness of the proposed method is successfully validated in real-world robotic tasks.

Abstract

Generative models, such as diffusion and flow-based models, have shown strong promise for robot policy learning by capturing complex and multimodal action distributions from demonstrations. However, policies trained solely with imitation learning often suffer from imperfect demonstrations and distributional shifts, while further improvement typically requires additional expert data. Reinforcement learning offers a natural solution through environment interaction, but effectively finetuning generative robot policies remains challenging due to the intractability of likelihood estimation. In this work, we propose RoboFFT, a forward-process reinforcement learning framework for finetuning generative robot policies, which applies forward noising to sampled actions and uses the weighted score / flow matching loss to construct a surrogate policy ratio for PPO-style updates. We evaluate RoboFFT with popular generative robot policies on representative simulation benchmarks, including long-horizon planning and sparse reward settings. Extensive experiments and analysis demonstrate that RoboFFT consistently improves performance while achieving better stability and training efficiency. We further integrate RoboFFT into a real world RL framework and demonstrate its effectiveness in real world tasks. Project website: https://student-of-holmes.github.io/RoboFFT/.

Problem

• generative robot policy
• reinforcement learning
• finetuning
• likelihood estimation
• imitation learning

Innovation

• Generative Robot Policy
• Forward Process Reinforcement Learning
• Surrogate Policy Ratio
• Flow Matching Loss
• Online Finetuning