Topology-Informed Visual Prompting For Vision Language Action Policies

2026-09-20

AI Summary

为解决复杂几何环境下的视觉-语言-动作任务，提出一种基于拓扑指导的视觉提示框架，利用模拟规划增强数据集，并通过视觉引导提高任务成功率。

Abstract

Vision-language-action (VLA) policies can struggle with manipulation tasks with complex obstacle geometries due to partial observability. These complex geometries can lead to similar visual observations or robot configurations requiring qualitatively different actions, a distinction that can be quantified using topological signatures. While motion planners with full knowledge of environment geometries and object states can reason about these signatures in planning, this information is often not known at deployment. To address this issue, we present a topology-guided visual-prompting framework that uses simulation-based planning to augment a nominal demonstration dataset and provides vision-based guidance at deployment. Our method uses a Gauss-Linking-Integral topological signature representation to capture important topological properties of the environment. Using privileged geometry information from a simulation approximation of our environment, we augment a VLA fine-tuning dataset with trajectories that move the system to a demonstrated signature and, from the new configuration, resume task execution. A vision-language model (VLM) is fine-tuned on the same dataset to both predict signatures from live camera observations and predict end-effector waypoints, which are rendered as visual prompts on the observations to guide the VLA. Across three simulated bimanual tasks and a real-world box pickup task, our method outperforms a VLA fine-tuned only on nominal demonstrations and a VLM-prompting baseline that can remove topology-relevant information from observations. On hardware, it exceeds the strongest baseline by 40% in task success. Project website: https://topology-vla.github.io.

Problem

• Topology
• Visual Prompting
• Vision-Language-Action Policies
• Partial Observability
• Complex Obstacle Geometries

Innovation

• Topology-Informed
• Visual Prompting
• Gauss-Linking-Integral
• Simulation-based Planning
• Vision-Language Model