The Low-Rank Structure of VLA Reinforcement Learning

2026-09-28

AI Summary

This study addresses the unclear mechanism by which reinforcement learning (RL) reshapes the policy of Vision-Language-Action (VLA) models. Leveraging flow-based VLA models and the LIBERO benchmark, combined with module replacement and linear probing techniques, this work systematically reveals for the first time that RL induces low-rank parameter updates within VLAs. These updates are found to concentrate in the timestep module, encoding discrete denoising signals and task relationships. Based on this insight, an offset vector-based policy steering method is proposed. The approach achieves 99.6% accuracy in predicting task success rates and effectively enhances policy performance through steering without requiring additional training.

Abstract

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including $π_{0.5}$ and GR00T~N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to $99.6\%$). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.

Problem

• Reinforcement Learning
• Vision-Language-Action Models
• Low-Rank Structure
• Post-training
• Interpretability

Innovation

• Vision-Language-Action Models
• Reinforcement Learning
• Low-Rank Updates
• Timestep Modules
• Shift Vector Steering