How Should Vision-Language-Action Models Use Proprioceptive State?

2026-08-03

AI Summary

This study addresses the lack of standardized practices in incorporating proprioceptive state information into vision–language–action (VLA) models, which obscures the practical benefits of such integration for closed-loop control, the role of historical states, and the optimal embedding location. Under fixed backbone architectures, training data, and evaluation protocols, the work systematically evaluates five state interface designs—discrete prompting, VLM prefix, action prefix, state expert, and feature modulation—and investigates the impact of history lengths ranging from 1 to 96 frames. Experiments span 45 atomic and 20 compositional tasks, yielding the first clear delineation of the effectiveness boundary of state inputs, the temporal utility of historical context, and the best integration strategy. The findings culminate in empirically grounded design principles that offer actionable guidance for developing state-aware VLA systems.

Abstract

Recent Vision-Language-Action (VLA) models almost universally take robot proprioceptive state as input, yet wire it in incompatible ways -- serialized into text prompts, projected into the vision-language prefix, or fed directly to the action expert -- and almost always as a single current frame. Three questions remain open: (1) whether, and on which tasks, current state actually improves closed-loop control; (2) how much state history helps, and whether its benefit reflects genuine temporal variation rather than added conditioning capacity; and (3) where state should enter the model -- the vision-language backbone or the action-generation module. We answer these questions through controlled experiments on a flow-matching VLA, fixing the backbone, training data, action representation, and evaluation protocol throughout. We implement five representative interfaces -- discrete state prompt, VLM prefix, action prefix, state expert, and feature modulation -- under matched implementation details, and evaluate them on 45 atomic tasks spanning three task families plus 20 composite tasks; we then sweep the state-history length from 1 to 96 frames to examine how historical state information affects model performance. The experiments yield systematic answers to all three questions, distilled into testable design principles for state-aware VLAs.

Problem

• Vision-Language-Action models
• proprioceptive state
• closed-loop control
• state history
• model architecture

Innovation

• Vision-Language-Action models
• proprioceptive state
• state history
• flow-matching
• closed-loop control