Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning

2026-05-28

AI Summary

This work addresses the inefficiency of vision-language-action (VLA) models in embodied reinforcement learning, which stems from sparse supervision and the difficulty of designing long-horizon reward functions. To overcome this, the authors propose Feat2Go, a framework that leverages a pretrained visual world model to automatically generate fine-grained, semantically staged progress targets through patch-level similarity and trend clustering. These targets yield an intrinsic reward signal without manual engineering, enabling structured value estimation by an embodied value model. Integrated with policy optimization algorithms such as PPO or GRPO, Feat2Go substantially improves task success rates—boosting out-of-distribution performance on ManiSkill3 from 17.5% to 82.9% for OpenVLA-OFT and achieving an average success rate of 88.8% on RoboTwin 2.0 domain-randomized tasks.

Abstract

Reinforcement learning is a promising approach for improving the capabilities of vision-language-action (VLA) models while avoiding the heavy data requirements of imitation learning. However, its effectiveness for VLA models is often constrained by sparse supervision and the difficulty of designing informative reward signals for long-horizon manipulation. In this work, we present Feat2Go, a fine-grained value estimation framework for embodied reinforcement learning. Specifically, Feat2Go first derives a continuous progress target from a pretrained visual world model by measuring patch-level similarity to subgoal states and partitioning episodes into semantic stages with trend-based clustering. We then train an embodied value model to predict this structural progress from the current observation and task instruction, and use the predicted value to reshape terminal rewards during policy optimization. The proposed framework is compatible with existing VLA policy reinforcement learning pipelines, including PPO and GRPO, and does not rely on manual reward engineering. Extensive experiments on ManiSkill3 and RoboTwin 2.0 demonstrate that Feat2Go consistently improves the performance of existing VLA models under both single-arm and bimanual manipulation settings. More specifically, on ManiSkill3, Feat2Go improves OpenVLAOFT from 17.5% to 82.9% average out-of-distribution success while retaining 96.9% in-distribution performance. On RoboTwin 2.0, Feat2Go achieves an average success rate of 88.8% in domain-randomized task settings, outperforming prior reinforcement learning methods.

Problem

• embodied reinforcement learning
• vision-language-action models
• sparse rewards
• long-horizon manipulation
• reward design

Innovation

• fine-grained value estimation
• visual feature grounding
• embodied reinforcement learning
• subgoal-aware reward shaping
• semantic stage clustering