Vision-TL-Action: Neuro-Symbolic Trajectory Generation from Visual Observations and Temporal Logic

2026-07-29

AI Summary

This work addresses the problem of generating long-horizon robot trajectories that satisfy temporal logic (TL) specifications using only multi-view visual observations and coordinate-free TL task descriptions, without requiring object geometric information during inference. The proposed approach integrates a neuro-symbolic architecture with bidirectional cross-attention to align TL syntax graphs with visual spatial tokens and condition a flow-matching trajectory generator. During training, predicate-to-region semantic grounding supervision enables joint learning of logical reasoning and visual perception. To the best of our knowledge, this is the first method to achieve vision-to-TL trajectory generation without geometric priors. It attains 67.45% Success@1024 on Panda tasks—surpassing the oracle-state baseline of 59.11%—and achieves 96.35% Success@256 on AntMaze, closely approaching oracle performance at 96.88%.

Abstract

Temporal logic (TL) provides a compositional language for the formulation of long horizon robotic tasks, but existing TL-conditioned trajectory generators can sidestep perception-to-symbol binding by encoding exact object geometry in the task graph. We introduce \emph{Vision-TL-Action}, which generates action trajectories from multi-view images, a coordinate-free TL syntax graph, and the robot initial state. TL-node tokens and spatial visual tokens are fused through bidirectional cross-attention, and the resulting representation conditions a flow-matching trajectory generator. Visual tokens are augmented only with normalized image-plane locations and camera-view identifiers, while a training-only predicate-to-region objective encourages grounding to referenced objects. Consistent with prior work in this domain, we evaluate the model using Success@$K$, the fraction of tasks for which at least one of K sampled trajectories satisfies the TL specification. On Panda task, our model achieves 67.45% Success@1024, compared with 59.11% for the oracle-state baseline. On AntMaze task, it achieves 96.35% Success@256, comparable to the oracle result of 96.88%. Resolution and intervention studies show that spatial detail depends on semantic grounding and predicate identity affects both attention and performance. These results demonstrate a direct mapping from visual observations and structured TL goals to action trajectories without requiring object geometry at inference. Code is available at https://github.com/AricLau07/vision-tl-action.

Problem

• Temporal Logic
• Vision-based Planning
• Neuro-Symbolic
• Trajectory Generation
• Semantic Grounding

Innovation

• neuro-symbolic
• temporal logic
• trajectory generation
• visual grounding
• cross-attention