Trajectory-Level Redirection Attacks on Vision-Language-Action Models

2026-06-11

AI Summary

This work addresses a critical trajectory-level safety vulnerability in existing vision-language-action (VLA) models, wherein attackers can induce complete deviations from intended action trajectories through semantically preserved, minimal perturbations to textual instructions. The study formally introduces the "instruction-preserving trajectory redirection" threat model, revealing the susceptibility of VLA systems to textual perturbations during closed-loop control. To exploit this vulnerability, the authors propose a policy-based online prompt search algorithm that efficiently generates adversarial prompts under semantic constraints by leveraging environment interaction and rollback mechanisms. Experimental results demonstrate that the method achieves high success rates in steering VLA models toward attacker-specified final task objectives across both simulated and real-world robotic platforms.

Abstract

Vision-language-action (VLA) policies bring natural language into closed-loop robot control, enabling robots to execute manipulation tasks directly from text instructions. The same interface gives text a recurring role in control because the prompt is reused at every replanning step, and each prompt-conditioned action changes the future observations on which the policy acts. Existing VLA attacks study adversarial prompts that elicit targeted low-level actions or make such actions persist across changing images. We identify a stronger trajectory-level failure mode: a prompt that still $\textit{appears}$ to specify the intended task but redirects the final physical outcome. We mathematically formalize this setting as $\textit{command-preserving trajectory redirection}$, a prompt-only threat model in which the attacker chooses one prompt before the episode, all policy and environment components remain fixed, and the prompt must stay close to the benign instruction while omitting target words and correction language. To find such prompts, we introduce an on-policy prompt search method that uses rollouts to discover perturbations whose closed-loop behavior tracks a target task while satisfying the command-preserving constraints. Experiments in simulation and on hardware show that near-benign prompt perturbations can redirect VLA rollouts to attacker-specified targets. These results expose a trajectory-level vulnerability in VLA instruction grounding: text that appears to preserve the intended command can still give an adversary control over the robot's final physical outcome. Project website: https://vla-redirection-attack.github.io/

Problem

• trajectory-level redirection
• vision-language-action models
• adversarial prompts
• command-preserving attack
• instruction grounding

Innovation

• trajectory-level attack
• vision-language-action models
• command-preserving redirection
• adversarial prompting
• closed-loop robot control