RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations

2026-09-20

AI Summary

为解决多机器人在部分可观测性下的协作问题，本文通过构建RoboTalk数据集和训练小型视觉语言模型来学习多模态演示中的通信与协调。

Abstract

Multi-robot collaboration could enable more efficient and scalable solutions to complex robotic tasks, but collaboration under partial observability remains challenging. Natural-language communication offers a promising approach to coordinating robots under partial observability. However, in decentralized manipulation, jointly learning explicit inter-robot communication and skill-level action selection from multimodal demonstrations remains underexplored for small vision-language models (VLMs) intended for on-device deployment. To address this gap, we introduce RoboTalk, a synthetic data-generation pipeline and dataset of 7,950 multimodal trajectories spanning 53 mobile-manipulation kitchen tasks for training small VLMs to communicate and coordinate. The dataset includes a leader-follower planning protocol, tool calls (perception, manipulation, navigation, and communication), rationale traces, and diversified natural-language communication. Fine-tuning open-source models on our dataset can reach 77% success on novel held-out tasks, a significant improvement over the untuned open source models, which had a success rate of around ~2%.

Problem

• multi-robot collaboration
• partial observability
• multimodal demonstrations
• inter-robot communication

Innovation

• synthetic data-generation pipeline
• multimodal demonstrations
• small VLMs
• inter-robot communication
• leader-follower planning