vision-language skill sequencing

This study addresses the vulnerability of Vision-Language-Action (VLA) models to visual shortcuts, which cause them to disregard instructions and struggle to generalize to unseen skill compositions. To overcome this limitation, this work proposes the CRAFT framework, which leverages counterfactual data augmentation to eliminate spurious visual correlations. Furthermore, it introduces a novel skill-representation-based cross-execution supervision transfer mechanism that facilitates effective skill decoupling and reliable reuse. Extensive evaluations across multiple simulation benchmarks and real-world robotic platforms demonstrate that the proposed approach significantly improves execution success rates for undemonstrated skill compositions while preserving high performance on demonstrated tasks. Consequently, this method effectively resolves the combinatorial generalization challenge inherent in VLA models.

研究探讨了通过激活空间中的向量表示来操纵大型语言模型的程序技能，发现这些技能可以作为方向被激活和组合，以实现更高级别的功能和个人化优化。

This study addresses the data scarcity and generalization bottlenecks encountered by small models in robotic manipulation by proposing an LLM-SLM orchestration framework based on task decomposition and skill composition. The core methodology leverages large language models to drive data synthesis for training small language models, designs a skill-aware context-free grammar (CFG) to constrain the action space, and introduces a progressive orchestration strategy to ensure reliable task planning and natural language instruction parsing. Experimental evaluations conducted on both unmanned aerial vehicles and ground vehicles demonstrate that the proposed approach significantly outperforms existing baselines, substantially enhancing the model's zero-shot generalization capability to unseen tasks.

Existing interactive robotic task planners suffer from poor generalization, reliance on predefined modules, and labor-intensive prompt engineering. Method: This paper proposes a long-horizon task planning framework for real-world scenarios, introducing a novel functional architecture that jointly integrates high-level language-based planning with low-level skill invocation, grounded in vision-language multimodal scene understanding. Contribution/Results: The framework enables dynamic goal interpretation, online replanning, and zero-shot cross-task transfer—requiring only lightweight task instructions without domain-specific fine-tuning or manual prompt design. Evaluated on a real-world bubble tea preparation task, it successfully generates and executes unseen goals, accommodates mid-task user requests for modification, performs precise online replanning, and generalizes to diverse household service scenarios.

Direct generation of PDDL goals or task sequences by large language models (LLMs) often yields semantically abstract, non-executable outputs, hindering integration with robot task and motion planning (TAMP). Method: We propose an LLM knowledge distillation framework that extracts object-level state-change knowledge via prompt engineering, constructs a function-oriented object network (FOON), and automatically compiles it into semantically aligned, executable PDDL subgoals. Contribution/Results: This FOON-PDDL joint representation establishes the first structured synergy between LLM-derived high-level semantics and classical planners’ action-object constraints. Evaluated on simulated pick-and-place tasks, our approach improves subgoal success rate by 37%, significantly enhancing planning feasibility and cross-task generalization.

为解决工具使用语言模型训练数据昂贵问题，提出技能预训练（SPT）方法，利用公开技能包作为训练数据，提高模型性能。

This study addresses the challenge faced by vision-language models (VLMs) in organizing discrete events from video streams into persistent skill libraries by proposing a streaming embodied skill discovery paradigm. Methodologically, it introduces the Counterfactual Library State Rebalancing (CLaRe) algorithm to optimize skill grouping, which, combined with supervised fine-tuning, enables automatic skill localization, transition, and reuse decision-making. To systematically evaluate these capabilities, the authors construct the Video2Skill benchmark. Experimental results reveal existing skill-scaling bottlenecks in current VLMs and demonstrate that accurately recognizing insufficient available skills constitutes a core challenge for general-purpose planning. Ultimately, this work provides a novel pathway toward constructing reusable embodied skill libraries.

This work addresses the challenge that existing vision-language-action models struggle to accurately execute natural language instructions involving spatiotemporal and logical constraints, while also lacking interpretability. The authors propose a hierarchical framework that, for the first time, deeply integrates Signal Temporal Logic (STL) between language understanding and robotic execution. The approach decomposes high-level instructions into subtasks and generates verifiable, optimizable, and correctable STL specifications, which dynamically schedule low-level policies. By combining vision-language models, STL, model predictive control, and learned policies, the method enables an end-to-end mapping from natural language instructions to formal specifications, supporting online monitoring and replanning. Experiments in real-world tabletop environments demonstrate significant improvements in accuracy, reliability, and interpretability of language-guided robotic tasks.

This study addresses the challenges of large control-level action search spaces, error accumulation, and reliance on manual annotations for symbolic skills in long-horizon planning. To this end, it proposes a flow matching-based latent world model coupled with a hierarchical planning framework. By designing four skill abstraction mechanisms that map noise seeds to symbolic labels, the method automatically generates skill actions without domain knowledge, enabling single-execution state transitions that effectively integrate representation learning with symbolic reasoning. Evaluated on simulated block rearrangement tasks, the proposed unsupervised approach maintains high success rates across multi-skill scenarios, validating the effectiveness of the introduced abstraction mechanisms.

This work addresses two prevailing paradigms in robot learning—“weight internalization” and “code-based skill self-generation”—by clarifying their distinctions and evolutionary trajectories, and tackling open challenges such as ambiguous skill definitions, ill-defined self-improvement mechanisms, and cross-platform adaptability. The authors propose a unified taxonomic framework that articulates five distinct interpretations of “skills,” operationalizes self-improvement for the first time, and systematically evaluates six technical families: vision-language-action models, zero-shot program synthesis, closed-loop self-repair, persistent skill memory, reinforcement learning–based skill discovery, and large language model–driven skill repositories with evolutionary search. A review of 77 representative systems reveals that only a few—including ASPIRE, ENPIRE, and RoboClaw—achieve open-ended self-improvement loops, while highlighting critical gaps in current commercial skill markets regarding adaptability and safety validation.