R2RI: A Multi-View Event and RGB Dataset for Robot-to-Robot Interaction

2026-10-05

AI Summary

This study addresses the lack of large-scale benchmark datasets for robot-to-robot interaction (R2RI) research by constructing the first multi-view, event-RGB multimodal dataset tailored to this domain. Comprising over 6.5 million frames and nearly 5,000 video sequences, the dataset integrates both first-person and third-person perspectives alongside a comprehensive spatiotemporal annotation framework designed to capture authentic social behavioral dynamics. Building upon this resource, the authors propose a dedicated evaluation benchmark that systematically compares performance disparities across modalities under state-of-the-art perception algorithms. By fully open-sourcing all data and annotations, this work provides essential infrastructure to advance research in collaborative systems and social robotics.

Abstract

Understanding and modeling interactions between autonomous agents is a fundamental challenge in robotics, with broad implications for collaborative systems, social robotics, and human-robot coexistence. Although the study of robot interactions has emerged as a compelling research direction, progress has been severely hampered by the absence of large-scale benchmarks. In this paper, we introduce Robot-to-Robot Interaction (R2RI), the first dataset specifically designed to address the Robot-Robot Interaction (RRI) task. R2RI consists of different humanoid robots and realistic interactions modeled on real human social behaviors. Complementary viewpoints are available, \textit{i.e.}, an egocentric perspective from each robot's onboard sensors, and an exocentric perspective from external fixed cameras, thus enabling rich spatial and contextual understanding of the interaction dynamics. The dataset comprises more than $6.5$M frames and $\approx5000$ videos at $120$ fps, including Event and RGB domains. We investigate pros and cons of each domain, comparing state-of-the-art approaches for a number of key sensing and interaction based tasks. We publicly release the dataset and its annotations for all tasks and modalities at https://github.com/MagriniGabriele/R2RI.

Problem

• Robot-to-Robot Interaction
• Multi-View Dataset
• Event Camera
• Autonomous Agents
• Benchmark

Innovation

• Robot-to-Robot Interaction
• Multi-View Dataset
• Event Camera
• Egocentric Vision
• RGB-Event Fusion