TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models

2026-08-07

AI Summary

This work addresses the vulnerability of existing vision–language–action (VLA) models to distribution shifts during downstream task adaptation and the inefficiency of conventional reinforcement learning (RL) fine-tuning, which uniformly updates all modules despite their functional heterogeneity. To overcome these limitations, the authors propose TEMPO, a novel framework that freezes the pretrained vision–language backbone during RL fine-tuning and instead decouples the optimization of the semantic projection layer and low-level action experts. TEMPO introduces a dual-timescale update mechanism: the semantic module is updated at a low frequency to preserve high-level semantic stability, while the action module is updated at a high frequency to rapidly adapt to control feedback. This approach achieves the first decoupled optimization of semantics and action in VLA models, significantly outperforming current VLA architectures and RL fine-tuning baselines on both the CALVIN benchmark and real-world robotic tasks, yielding higher and more stable online rewards.

Abstract

Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically apply a single, uniform update strategy to all model components, ignoring their distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations, and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. We update them at different rates--the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing high-level semantic representations while still allowing the action expert to learn efficiently from online feedback. Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on two real-world tasks.

Problem

• vision-language-action models
• reinforcement learning post-training
• semantic-action decoupling
• distribution mismatch
• policy update stability

Innovation

• semantic-action decoupling
• two-timescale RL
• vision-language-action models
• RL post-training
• action expert