When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models

2026-09-30

AI Summary

This study addresses the generalization bottleneck of Vision-Language-Action (VLA) models under counterfactual instructions, which arises from the failure of instruction-action binding. By integrating behavioral analysis with internal state probing techniques, we elucidate the underlying mechanisms of this limitation. Accordingly, we propose Equivariant Counterfactual Training (ECT), a novel framework that optimizes imitation learning objectives through dual strategies of data augmentation and loss regularization, effectively encouraging the model to fuse visual and linguistic information for correct action selection. Experimental results demonstrate that our approach significantly enhances generalization to unseen spatial configurations, achieving success rates of up to 88% on both the LIBERO-PRO benchmark and a physical UR5e robotic platform.

Abstract

Vision-language-action (VLA) models can exceed 90% success on in-distribution tasks and withstand nuisance changes that preserve the required action, yet fail under counterfactual changes that demand a different action. Aggregate robustness scores can therefore conceal a more specific failure, in which a policy responds to both language and vision yet does not combine them to select the action the task requires. We call this failure instruction-action binding. Instructions cue familiar trajectory families, and visual feedback adjusts their execution. Behavioral analyses of fine-tuned $π_{0.5}$ and GR00T-N1.7 policies reveal that failed rollouts often retain the source behavior or switch to another demonstrated task. These switches show that language is not simply ignored. Readouts and interventions connect these choices to task-conditioned internal states. Our analysis of the imitation objective shows how narrow conditional action support can leave grounded and instruction-keyed solutions indistinguishable on the demonstrations. This motivates Equivariant Counterfactual Training (ECT), which acts at two levels. ECT data supply valid demonstrations in which the same instruction requires different actions in distinguishable scenes, while the ECT loss trains each demonstration with its counterpart in the same update. In a controlled LIBERO-PRO comparison, full ECT raises $π_{0.5}$'s mean position-swap success from 36% to 59%. On CALVIN, where counterparts already occur in the original data, the ECT loss improves five-task completion without new demonstrations. On a real UR5e under a fixed demonstration budget, full ECT raises unseen-position success from 8% to 88%.

Problem

• Vision-Language-Action models
• generalization failures
• instruction-action binding
• counterfactual changes
• robotics

Innovation

• Vision-Language-Action Models
• Instruction-Action Binding
• Equivariant Counterfactual Training
• Counterfactual Generalization
• Imitation Learning