RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation

2026-06-11

AI Summary

This work addresses the limited fine-grained understanding of task execution processes in current vision-language models for robotic manipulation, which hinders accurate assessment of physical states and temporal progress. To bridge this gap, the study formally defines and quantifies “process awareness” for the first time, introduces ProcessData—a dataset of 58k question-answer pairs grounded in physically realistic execution trajectories—and proposes RoboProcessBench, a benchmark comprising 12 diagnostic question categories that evaluate model performance along static monitoring and dynamic reasoning dimensions. Supervised fine-tuning and evaluation of models such as Qwen2.5-VL-7B and InternVL-3-8B reveal pervasive deficiencies in process understanding; however, fine-tuning with ProcessData-SFT yields significant improvements across key dimensions including local state, motion, progress, and primitive-level perception.

Abstract

Vision-language models (VLMs) are increasingly explored as visual critics, reward generators, and failure detectors in robotic manipulation. These roles implicitly require models to judge not only final task success, but also how a manipulation execution is physically and temporally progressing. However, existing evaluations fail to test whether VLMs possess fine-grained process understanding. To address this gap, we present RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation. RoboProcessBench decomposes such capability into two complementary dimensions, \emph{static monitoring} and \emph{dynamic reasoning}, instantiated as 12 diagnostic question families covering phase, contact, motion, coordination, primitive-local progress, temporal order, outcome, and primitive-level transitions. Built from physically grounded execution traces, the curated benchmark corpus ProcessData contains \textasciitilde 58k question-answer pairs across 260 manipulation tasks, which is further split into ProcessData-SFT and ProcessData-Eval for post-training and evaluation purposes. Extensive evaluation of various VLMs on ProcessData-Eval reveals broad limitations across 12 diagnostic task families, suggesting current models still lack robust process-aware understanding of manipulation executions. But with ProcessData-SFT, the post-trained \textit{Qwen2.5-VL-7B} and \textit{InternVL-3-8B} exhibit consistent gains on local state, motion, progress, and primitive-aware cues. These results demonstrate that RoboProcessBench serves as both an evaluation benchmark and a learnable supervision source for developing VLMs capable of monitoring and evaluating robotic manipulation processes. Project webpage: \href{https://processbench-2026.github.io/RoboProcessBench-Web/}{https://processbench-2026.github.io}.

Problem

• process-aware understanding
• vision-language models
• robotic manipulation
• benchmarking
• execution monitoring

Innovation

• process-aware understanding
• vision-language models
• robotic manipulation
• benchmarking
• execution trace