Task Robustness via Re-Labelling Vision-Action Robot Data

2026-06-09

AI Summary

Existing robotic datasets often lack sufficient linguistic and action diversity, limiting the generalization of policies to varied instructions. This work proposes TREAD, a framework that leverages pretrained vision-language models (VLMs) to automatically re-annotate existing robot demonstration data through a three-stage pipeline—semantic subtask parsing, video segmentation, and generation of diverse, object-attribute-rich instructions—without requiring additional data collection. By enriching the dataset with high-quality language-action pairs, TREAD enables more robust training of language-conditioned policies. Experiments on the LIBERO benchmark demonstrate that policies trained with TREAD-augmented data significantly outperform baselines on unseen tasks and novel targets, effectively enhancing the generalization capabilities of language-guided policy learning and planning.

Abstract

The recent trend in scaling models for robot learning has resulted in impressive policies that can perform various manipulation tasks and generalize to novel scenarios. However, these policies continue to struggle with following instructions, likely due to the limited linguistic and action sequence diversity in existing robotics datasets. This paper introduces Task Robustness via Re-Labelling Vision-Action Robot Data (TREAD), a scalable framework that leverages large Vision-Language Models (VLMs) to augment existing robotics datasets without additional data collection, harnessing the transferable knowledge embedded in these models. Our approach leverages a pretrained VLM through three stages: generating semantic sub-tasks from original instruction labels and initial scenes, segmenting demonstration videos conditioned on these sub-tasks, and producing diverse instructions that incorporate object properties, effectively decomposing longer demonstrations into grounded language-action pairs. We further enhance robustness by augmenting the data with linguistically diverse versions of the text goals. Evaluations on LIBERO demonstrate that policies trained on our augmented datasets exhibit improved performance on novel, unseen tasks and goals. Our results show that TREAD enhances both planning generalization through trajectory decomposition and language-conditioned policy generalization through increased linguistic diversity.

Problem

• instruction following
• linguistic diversity
• action sequence diversity
• robot learning
• task generalization

Innovation

• Vision-Language Models
• Data Augmentation
• Instruction Diversification
• Trajectory Decomposition
• Robot Learning