FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

2026-06-03

AI Summary

This work addresses the challenge of deploying vision-language-action (VLA) models on real robots, where the absence of effective reward signals limits performance. To overcome this, the authors propose FlowPRO, a reward-free offline reinforcement fine-tuning framework. FlowPRO leverages an intervention-rollback mechanism to generate positive-negative trajectory pairs and employs smooth interpolation and batch mixing to provide dense, state-level supervision for flow-matching VLAs. Furthermore, it introduces the RPRO algorithm, which integrates contrastive optimization with an explicit proximal regularizer to anchor the absolute scale of implicit rewards, thereby mitigating reward hacking issues observed in Flow-DPO. Evaluated on four long-horizon bimanual tasks, FlowPRO significantly outperforms four baseline methods, and ablation studies confirm the contribution of each proposed component.

Abstract

Post-training Vision-Language-Action (VLA) models into policies that can be reliably deployed on real robots remains a major bottleneck. SFT and DAgger exploit failure signals only indirectly, and reward-based RL is bottlenecked by the difficulty of real-world reward design and of training reliable critics. We present FlowPRO, a reward-free offline reinforced fine-tuning framework for flow-matching VLAs. Algorithmically, we propose RPRO (Robotic Flow-matching Proximalized Preference Optimization), a preference-optimization objective tailored to the flow-matching action head of VLA models. RPRO pairs a contrastive optimizer with an explicit proximal regularizer that anchors the absolute magnitude of the implicit reward, thereby eliminating the reward-hacking failure mode of plain Flow-DPO. On the data side, a teleoperated intervention-and-rollback paradigm produces naturally paired positive and negative trajectories $(τ^w, τ^l)$ on a real robot from a single operator action; a Smooth Interpolation procedure, combined with batch mixing, then converts these sparse corrections into dense per-state supervision while preserving the base policy's capabilities. On four long-horizon bimanual tasks, FlowPRO attains the highest success rate, outperforming four representative baselines, and ablations confirm the contribution of each loss component.

Problem

• Vision-Language-Action
• reward-free reinforcement learning
• robotic policy fine-tuning
• flow-matching
• preference optimization

Innovation

• Flow-Matching
• Preference Optimization
• Reward-Free RL
• Proximal Regularization
• Teleoperated Intervention