RoboChrono: A Real Robot Benchmark for Streaming Task Understanding

2026-09-28

AI Summary

This study addresses the persistent challenge of correlating visual observations with interaction histories and task progress in robotic manipulation. To this end, it constructs a streaming task understanding benchmark grounded in real-world robot and human demonstrations, encompassing seven task categories such as recognition, alignment, and temporal localization. Moving beyond aggregate scoring, the work introduces a diagnostic evaluation perspective to conduct zero-shot assessments and input ablation studies across eighteen vision-language models. The findings reveal that strong visual matching capabilities do not inherently translate to temporal ordering proficiency, quantifying the performance gap between them. Furthermore, the analysis demonstrates that next-action prediction relies predominantly on priors rather than real-time visual evidence. Collectively, this research establishes a novel paradigm for evaluating specific model capabilities in embodied AI contexts.

Abstract

Understanding ongoing robot manipulation requires models to interpret visual observations in relation to interaction history and task progress. We introduce RoboChrono, a benchmark for streaming task understanding comprising 39 scenarios and 34,713 evaluation instances, constructed from real robot executions and complementary bare-hand human recordings. The benchmark evaluates seven tasks grouped into recognition, alignment, and temporal grounding, covering action understanding and anticipation, visual correspondence, temporal ordering, and action localization. Zero-shot evaluation of 18 vision-language models reveals substantial differences across tasks. GPT-6-Astra achieves 98.3% accuracy on Frame Matching but 68.3% on Frame Ordering, while RynnBrain1.1-122B-A10B exhibits a larger gap, reaching 95.4% and 32.9%, respectively. Input ablations on matched questions with five open-weight models further reveal distinct dependencies on visual evidence: removing visual observations reduces Current Action Recognition accuracy by 22.1 percentage points, whereas Next Action Prediction decreases by only 0.7 points. These findings show that strong visual matching does not consistently coincide with strong temporal ordering, and suggest that next-action prediction can be supported by task and action priors even when visual evidence is unavailable. RoboChrono provides a diagnostic setting for examining these differences, highlighting the need for capability-specific evaluation beyond aggregate scores when assessing task understanding in robot manipulation.

Problem

• streaming task understanding
• robot manipulation
• vision-language models
• temporal grounding
• benchmark evaluation

Innovation

• streaming task understanding
• robot benchmark
• vision-language models
• temporal grounding
• input ablation