In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

2026-08-06

AI Summary

This work addresses the limitations of existing vision–language–action (VLA) models, which rely on behavioral cloning and struggle to integrate language reasoning with low-level control, often compromising action accuracy and real-time performance due to unconstrained language generation. The authors propose a novel VLA training paradigm that applies supervision solely at the action level and prioritizes language understanding over generation. By leveraging contextual post-training and an agent-based tool-calling mechanism—including open-vocabulary detection, monocular depth estimation, and vision–language models—the approach enables structured context injection and evidence-conditioned spatial description generation. Evaluated on RoboCasa-GR1, SimplerEnv, LIBERO simulation benchmarks, and eight real-world robotic tasks, the method significantly outperforms current chain-of-thought approaches, achieving state-of-the-art performance in both effectiveness and efficiency.

Abstract

Vision-Language-Action (VLA) models have become the dominant recipe for generalist manipulation, yet they are almost universally trained by behavior cloning: a policy imitates expert action chunks conditioned on a static image and a fixed instruction. A natural remedy is to inject explicit reasoning through textual chain-of-thought (CoT). We show, both empirically and analytically, that free-form textual CoT degrades low-level control: the reasoning it produces is ungrounded, its latency breaks closed-loop timing, and, crucially, the reasoning and action tokens are optimized against conflicting objectives so that the policy learns to narrate rather than to act. We argue that what a VLA needs is not the ability to generate language, but the ability to consume grounded language. To this end we introduce \textbf{\ourmethod{}}, a framework that endows a VLA with language competence through (i) in-context post-training, in which perceptual evidence is injected as structured context and the model is supervised only on actions, and (ii) an agentic tool-use interface, in which the policy queries open-vocabulary detectors, monocular depth, and a vision--language model to actively acquire task-relevant information. Rather than emitting a single templated caption, our data engine produces diverse, paraphrased, and evidence-conditioned spatial descriptions, so that the policy learns to interpret language it has never seen verbatim. Across the RoboCasa-GR1, SimplerEnv, and LIBERO simulation benchmarks, together with 8 real-world robot manipulation tasks, our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches under matched configurations.

Problem

• Vision-Language-Action
• In-Context Learning
• Language Grounding
• Robot Manipulation
• Chain-of-Thought

Innovation

• In-Context Learning
• Vision-Language-Action Models
• Agentic Tool Use
• Grounded Language Understanding
• Post-Training