Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking

2026-08-04

AI Summary

This work addresses the vulnerability of vision-language-action (VLA) robotic systems to physical adversarial patch attacks, which hijack policy-critical action-visual attention and lead to task failure. The study is the first to identify and formally name this “attention hijacking” mechanism. To mitigate it, the authors propose SARF (Structure-Aware Robust Fine-tuning), a zero-inference-overhead method that fine-tunes only the visual encoder through feature anchoring, critical attention correction, and language-guided geometric consistency constraints. Evaluated on the LIBERO benchmark, SARF reduces the attack-induced failure rate of OpenVLA from 100% to 28.6%. On a real PiPER robotic arm, it improves task success under attack from 23.0% to 65.0% without degrading performance on clean inputs, demonstrating strong cross-task and cross-architecture transferability.

Abstract

Vision-Language-Action (VLA) policies promise general robotic manipulation, but their robustness against physical-world attacks remains fragile. In particular, we show that physically realizable adversarial patches can reliably induce failures by triggering a mechanism we call policy-critical action-to-vision attention hijacking, where action-conditioned attention is diverted from task-relevant regions to a localized patch. To demonstrate the threat, we propose Attention-Guided Semantic Disruption (AGSD), an Expectation-over-Transformation (EOT) optimized printable patch that jointly (i) concentrates action-to-vision attention on the patch and (ii) disrupts vision-language semantic alignment, yielding strong cross-task and cross-architecture transfer. To mitigate such attacks, we introduce Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead defense that fine-tunes only the visual encoder using feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions. On LIBERO, SARF reduces OpenVLA's failure rate under AGSD from 100% to 14.2%-56.8% (28.6% average) across suites while preserving clean performance, and on a real PiPER manipulator it improves average success under AGSD from 23.0% to 65.0%. These results highlight mechanism-level robustness as a practical path to securing VLA robots against physical attention hijacking.

Problem

• Vision-Language-Action
• adversarial patches
• attention hijacking
• physical-world attacks
• robustness

Innovation

• Structure-Aware Robust Fine-Tuning
• Attention Hijacking
• Vision-Language-Action
• Adversarial Patch Defense
• Feature Anchoring