From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom

2026-09-28

AI Summary

This study addresses the challenge of mapping qualitative semantics in natural language instructions to quantitative states for continuous control, as well as the issue of unintended constraints introduced by fully specified poses. To this end, we propose a typed semantic-geometric interface compiler architecture. This approach leverages conformal geometric algebra to distinguish task-relevant constraints from preserved degrees of freedom, decoupling semantic topology from geometric specifications, and generates composable continuous control objectives via RMPflow compilation. Evaluations in MuJoCo simulations and on a Franka robotic arm demonstrate that the proposed method significantly reduces critical error rates. Furthermore, closed-loop experiments confirm that fixing irrelevant degrees of freedom induces 20–71 mm error growth, validating the effectiveness of explicitly preserving the null space for precise alignment.

Abstract

Natural-language manipulation instructions specify qualitative relations, whereas continuous controllers require state-evaluable task quantities, differentials, and completion conditions. Because a qualitative relation generally leaves part of the relative configuration unspecified, expanding it into a complete pose can introduce unintended constraints. We present a typed semantic-to-geometric interface in which language specifies entities, relations, and phases, while each relation indexes a registered specification of its task-relevant distinctions and preserved freedoms. A robot-side compiler grounds these specifications, constructs relation-specific task maps and consistent differentials using conformal geometric algebra, and composes the resulting policies through RMPflow. To evaluate the division of responsibility between the language model and the compiler, we compared a Semantic Topology interface with one that additionally requires relation-specific geometric specifications over 60 instructions. Both produced correct shared semantic content in 41/60 cases, but critical errors under their respective interface requirements occurred in 19/60 and 58/60 cases. Across 64 grounded evaluations spanning eight geometric relation forms, the task maps preserved registered null directions and responded to relation-relevant perturbations; analytic directional derivatives agreed with finite differences, and Jacobian ranks matched the registered dimensions. In three closed-loop ablations using a simulated Franka Emika Panda in MuJoCo, fixing a relation-preserved coordinate increased median terminal progress error by 20.24--71.00~mm while the retained relation errors remained within their evaluation bounds. These results support compiling relation-visible geometry and preserved freedom together into composable continuous objectives.

Problem

• natural language instructions
• semantic-to-geometric grounding
• task-relevant freedom
• robot manipulation
• continuous control

Innovation

• Semantic-to-Geometric Interface
• Conformal Geometric Algebra
• RMPflow
• Task Maps
• Preserved Freedom