FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

2026-06-24

AI Summary

Vision-Language-Action models are often hindered by suboptimal imitation data, and existing reinforcement learning fine-tuning approaches suffer from low sample efficiency, unstable Q-functions, and poor exploration quality. This work proposes FORCE, a three-stage framework that innovatively integrates on-policy rollout–based value calibration pretraining with a self-distillation mechanism. The approach first mitigates Q-function distribution shift through value calibration and then leverages the calibrated Q-function to filter both policy-generated actions and expert demonstrations, retaining only high-value samples for policy updates. Requiring no human intervention, FORCE achieves an absolute 79% improvement in task success rate over baseline methods—surpassing current RL approaches by 10%—while accelerating training by 32.5%. It also effectively prevents performance degradation, substantially enhancing both training stability and efficiency in both simulated and real-world tasks.

Abstract

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.

Problem

• Vision-Language-Action
• Reinforcement Learning
• sample inefficiency
• catastrophic unlearning
• exploration data

Innovation

• Value-Calibrated Warm-Up
• Self-Distillation
• Vision-Language-Action
• Reinforcement Learning Fine-Tuning
• Sample Efficiency