RL Bootstrapping of OpenVLA-OFT for a Novel Robot Embodiment

2026-08-02

AI Summary

This work addresses the challenge of scarce demonstration data for emerging robotic platforms, such as cable-driven parallel robots, by proposing a language-conditioned policy transfer method that requires no expert demonstrations. Building upon the OpenVLA-OFT architecture, the approach fine-tunes policies in simulation through a two-stage reinforcement learning process: first employing PPO to learn basic directional motion primitives, then extending to object-conditioned instruction spaces via GRPO, augmented with dense geometric rewards and a simulation-to-policy training pipeline. This method achieves, for the first time, fully zero-shot language-aligned policy transfer, improving task success rates on four general instructions from 34.25% to 53.50% and attaining a strict success rate of 9.75% on the novel “move to <object>” instruction, demonstrating robust approaching behavior.

Abstract

Adapting a pretrained vision-language-action (VLA) policy to a new robot usually assumes embodiment-specific demonstrations. This assumption is especially restrictive for custom robots whose morphology differs strongly from the manipulators seen in large robot datasets. We study a harder setting: zero-demo embodiment alignment of OpenVLA-OFT on a cable-driven parallel robot (CDPR) with a simple gripper and a previously unseen control interface. Instead of supervised fine-tuning, we use reinforcement learning in simulation with dense geometric rewards computed from simulator state. The training is performed in two stages: a PPO stage for directional motion primitives, followed by GRPO continuation from the PPO checkpoint with an expanded instruction space that includes object-conditioned commands. On the four shared directional instructions, the average held-out success rate improves from 34.25\% after PPO to 53.50\% after PPO$\rightarrow$GRPO, with especially large gains on \texttt{move left} and \texttt{move backward}. In the GRPO stage we additionally introduce \texttt{move to <object>} over eight target objects and obtain 39/400 = 9.75\% strict success, while qualitative rollouts frequently show correct target-directed approach behavior before late-stage instability. Compared with prior OpenVLA and OpenVLA-OFT results, which rely on demonstration datasets and mostly standard rigid-arm embodiments, our method uses no embodiment-specific dataset at all. The results do not yet establish robust manipulation, but they provide stronger evidence that RL-only bootstrapping can create the first usable language-conditioned controller for a genuinely novel embodiment.

Problem

• robot embodiment
• zero-shot adaptation
• vision-language-action policy
• reinforcement learning
• cable-driven parallel robot

Innovation

• reinforcement learning
• zero-shot embodiment transfer
• vision-language-action models
• cable-driven parallel robot
• GRPO