Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

2026-08-05

AI Summary

This work addresses a critical limitation in existing vision-language-action (VLA) models, which fail to align spatial representations with the target object specified by language instructions, leading to degraded performance in occluded and fine-grained manipulation tasks. To resolve this, the authors propose an instruction-aware 3D understanding approach that first identifies the instruction-referred object, generates its three orthogonal views, and extracts features using VAE and VGGT encoders. The VLA model’s latent representations are then aligned specifically with these object-centric features, ensuring spatial alignment focuses on the instructed target rather than the entire scene. This method introduces, for the first time, an instruction-aware mechanism combined with a lightweight 345M-parameter backbone, achieving 93.9% success on the LIBERO benchmark, 4.47 on CALVIN, and a 54% average success rate on real-robot occlusion tasks—outperforming the best instruction-agnostic baseline by 32 percentage points.

Abstract

Recent Vision-Language-Action (VLA) methods improve generalization by aligning their representations with 3D scene geometry. However, these methods are fundamentally instruction-agnostic: the representations align the entire scene uniformly, neglecting the 3D geometry of the specific target object designated by the language instruction. This causes failures on fine-grained manipulation and target occlusion tasks, where success depends on accurate 3D understanding of the target object rather than the entire scene. To address this, we present Mind-VLA, an instruction-aware spatial representation alignment method for VLA models. Specifically, Mind-VLA first obtains the target object specified by the language instruction, then prepares its target-object tri-view and extracts the corresponding VAE and VGGT features. Finally, the latent representation of the VLA model is aligned with these features to enable instruction-aware 3D understanding. Mind-VLA reaches 93.9% on LIBERO and 4.47 on CALVIN with a compact 345M-parameter backbone. On real-robot tasks with target occlusion, Mind-VLA reaches 54% average success, outperforming the best-performing instruction-agnostic method in real-robot comparison by 32 percentage points. Code will be publicly available.

Problem

• Vision-Language-Action
• instruction-aware
• 3D scene geometry
• target object
• spatial representation alignment

Innovation

• instruction-aware
• spatial representation alignment
• vision-language-action
• 3D scene understanding
• target-object alignment