Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation

2026-10-03

AI Summary

This study addresses the challenge in robotic manipulation where current observations are insufficient to determine the state, and existing models struggle to leverage historical interactions for inferring task semantics. To this end, we introduce GiT, a multi-source dataset and evaluation benchmark encompassing scenarios such as biological laboratories. Methodologically, we incorporate counterfactual task pairs and fine-grained subtask annotations to precisely assess the utilization of historical information. The dataset integrates real-robot trajectories, UMI demonstrations, and ManiSkill simulations to facilitate the exploration of Vision-Language-Action (VLA) models. Evaluations reveal that current VLA models remain significantly deficient in manipulation tasks requiring historical context. Ultimately, this work provides critical data support and an evaluation paradigm for advancing history-aware robotic manipulation.

Abstract

Robotic manipulation often requires inferring task-relevant states from past interactions when the current observation alone is insufficient to determine the appropriate action. Despite progress in benchmarking memory-augmented vision-language-action (VLA) models, application-oriented tasks requiring history-dependent semantic inference remain underrepresented. We introduce GiT (Grounded in Time), a dataset and benchmark for grounding manipulation decisions in past events across biolaboratory, household, and industrial scenarios. It includes real-robot and Universal Manipulation Interface (UMI) style demonstrations covering 18 bimanual tasks, together with simulation data and a ManiSkill-based evaluation suite covering nine tasks. Fine-grained subtask annotations and annotated counterfactual task pairs, in which similar current observations require different actions depending on prior events, support policy learning and targeted evaluation of history use. Evaluations of representative end-to-end VLA models in simulation and on selected real-world tasks reveal substantial room for improvement in history-dependent manipulation. The dataset and benchmark are available at the project page.

Problem

• Temporal Grounding
• Robotic Manipulation
• Vision-Language-Action Models
• History-dependent Inference
• Benchmark

Innovation

• Temporal Grounding
• Vision-Language-Action Models
• Counterfactual Task Pairs
• Robotic Manipulation
• Multi-Source Benchmark