Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

2026-06-10

AI Summary

This work addresses the instability of existing vision-language-action (VLA) models in responding to semantically similar instructions and their limited capacity to fully leverage prompting. The authors propose a framework that operates without fine-tuning frozen VLA models by interactively searching for effective language sequences, distilling them into a test-time Language Feedback Policy (LFP), and incorporating a conformal calibration-enhanced head to determine when safe intervention is warranted. This approach achieves, for the first time, zero-shot, harmless language guidance for arbitrary frozen VLA models, offering strong safety guarantees in out-of-distribution scenarios and enabling recovery-oriented behavior generation. Experiments demonstrate a 24.7% improvement in simulated task success rates in seen environments and a 65.0% gain in real-world hardware trials, while maintaining robustness and harmlessness under both visual and semantic perturbations.

Abstract

Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone. As a result, both human instructions and zero-shot language models can fail to reliably steer VLAs toward successful task execution. In this work, we propose a framework that interactively searches for language sequences that improve closed-loop VLA task performance, distills these sequences into a test-time language feedback policy (LFP), and learns an improvement head that predicts when language steering will improve performance. We conformalize this improvement head to prevent harmful steering interventions, where the LFP decreases task performance relative to the original instruction on out-of-distribution scenarios. Crucially, our approach operates on arbitrary frozen pre-trained VLAs, requiring neither access to the original training distribution nor fine-tuning of the underlying model. On seen environments, our conformalized LFP improves base VLA performance by 24.7% in simulation and 65.0% in hardware. On visual and semantic perturbations, our conformalized LFP has strong harmlessness guarantees, and produces recovery behaviors not observed with open-loop prompting.

Problem

• Vision-Language-Action
• language grounding
• instruction following
• robot control
• harmless steering

Innovation

• Vision-Language-Action
• language feedback policy
• conformalized steering
• harmlessness guarantee
• frozen VLA