TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models

2026-06-02

AI Summary

This work addresses the performance degradation of vision-language-action (VLA) models under distribution shift during deployment by proposing a test-time training framework that introduces Latent Prompt Optimization (LPO) to embodied intelligence for the first time. During training, the method learns tunable latent prompts via proxy tasks; at test time, it adapts these prompts using only self-supervised signals without modifying the policy network. Experiments on SimplerEnv demonstrate that this approach significantly improves task success rates in both single- and multi-agent settings. The gains stem primarily from correcting critical decision points rather than globally adjusting policy behavior, thereby effectively enhancing the model’s generalization to novel environments.

Abstract

Vision-Language-Action (VLA) models trained on large-scale data have made remarkable progress, but they remain vulnerable to distribution shifts at deployment time. Recent VLA models suggest that prompts can serve as an efficient interface for steering policy behavior, but existing prompt-based steering typically relies on external guidance. This raises a natural question: can test-time training (TTT) for VLA be achieved by optimizing a prompt, so that the steering interface itself can be learned and adapted from interaction? We address this question with TTT-VLA, a test-time training framework based on Latent Prompt Optimization (LPO). During training, the latent prompt is learned with an additional proxy task, providing an extra learned conditioning signal for policy learning. At test time, TTT is performed by collecting interaction data from the current environment and optimizing only the latent prompt on those data using the proxy task's self-supervised signal, without modifying the policy itself. Experiments on SimplerEnv demonstrate that the proposed method consistently improves task success rates in both single- and multi-embodiment settings. Further analysis shows that the gains arise primarily from correcting a small number of critical decisions rather than globally altering policy behavior. These results suggest that LPO provides an effective and practical pathway for deployment-time improvement of foundation manipulation policies.

Problem

• Vision-Language-Action models
• distribution shift
• test-time training
• prompt optimization
• deployment adaptation

Innovation

• Test-Time Training
• Latent Prompt Optimization
• Vision-Language-Action Models
• Self-Supervised Adaptation
• Policy Steering