SSC: A Verifiable Structured Representation for Bimanual Manipulation Labelling

2026-08-05

AI Summary

Existing dual-arm robotic manipulation annotation methods struggle to balance the readability of natural language with the verifiability of templated formats, often leading to difficulties in automated validation or semantic over-segmentation. This work proposes the Structured Subtask Chain (SSC) framework, which represents manipulations as sequences of structured subtasks comprising action cores, flexible conditions, base motions, and post-state scene graphs. The SSC framework enables natural language rendering, rule-based automatic logical verification via state-transition constraints, and query-driven field completion. It is the first approach to unify readability and verifiability in manipulation annotations. Evaluated on BEHAVIOR-1K across 13 vision-language models, the framework effectively identifies annotation anomalies and completes missing content.

Abstract

Subtask labels decompose a long-horizon manipulation demonstration into shorter semantic segments for policy training and evaluation. Natural language descriptions are easy to read, but their linguistic variability makes automatic verification difficult. Rigid template formats, such as BEHAVIOR-1K's skill_annotation, are linguistically over-segmented, hindering both readability and annotation consistency. We propose the Structured Subtask Chain (SSC), a state-transition representation that bridges these extremes. A demonstration is a sequence of Structured Subtask Template (SST) entries. Each SST stores core action components (subject, predicate, object), flexible conditions (adverbial modifiers such as spatial or instrumental phrases), a base-motion field separate from arm actions, and an after-state scene graph. Built on this format, SSC supports three vision-language assisted functions: rendering SSTs as natural language, checking the assembled chain against four state-transition rules, and completing underspecified fields through a query resolution cascade. We instantiate the pipeline on BEHAVIOR-1K (50 tasks, 3 episodes per task, 2,357 annotated action cells) for logic verification and content completion, evaluating 13 selected state-of-the-art VL models as candidate verifiers and reporting labelling anomalies.

Problem

• bimanual manipulation
• subtask labelling
• structured representation
• natural language variability
• annotation consistency

Innovation

• Structured Subtask Chain
• bimanual manipulation
• state-transition representation
• vision-language verification
• action annotation