LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratories

2026-07-26

AI Summary

This work addresses the challenge of reliably handling irreversible, safety-critical experimental failures in chemical self-driving laboratories, where robots lack fine-grained failure data and standardized evaluation protocols. The study introduces the first fine-grained (11-class) robotic failure analysis framework tailored for chemical experiments, comprising a simulation environment with controllable multi-level (control, physical, semantic) fault injection, a large-scale multidimensional failure trajectory dataset, a six-dimensional capability evaluation benchmark, and a dedicated vision-language diagnostic model. A structured diagnosis and recovery instruction generation mechanism enables closed-loop fault tolerance. The fine-tuned vision-language model achieves 92.58% failure detection accuracy and 85.58% temporal localization accuracy in seen environments, and when deployed as a real-time supervisor, it improves downstream task success rates by 10–20 percentage points.

Abstract

The deployment of embodied agents in self-driving laboratories could accelerate scientific discovery, yet their reliability is constrained by the irreversible and safety-critical nature of chemical experiments. Progress is further hindered by scarce failure data and the lack of fine-grained evaluation protocols. To address these challenges, we introduce LabRobFail, a failure-centric framework for learning and evaluating robotic failure analysis in chemical laboratories. LabRobFail-Sim injects controllable failures at the control, physics, and semantic levels, enabling the construction of LabRobFail-Data, which contains over 20,000 trajectories across 70+ task scenarios, five failure categories, and 11 fine-grained failure types. LabRobFail-Bench evaluates six capabilities spanning task understanding, failure detection, temporal localization, severity assessment, failure classification, and actionable correction. We further develop LabRobFail-VLM, a domain-specialized vision-language model that generates structured failure diagnoses and recovery instructions. On seen environments, it achieves 92.58% failure-detection accuracy and 85.58% temporal-localization accuracy, substantially outperforming general-purpose VLMs. When integrated as a real-time supervisor, it improves downstream VLA task success rates by 10-20 percentage points, demonstrating the value of fine-grained failure understanding for closed-loop recovery and reliable laboratory autonomy. Our code and data are available at https://github.com/Su-ISE-2001/SciRobo

Problem

• robotic failure analysis
• self-driving laboratories
• chemical experiments
• failure data scarcity
• evaluation protocols

Innovation

• failure analysis
• self-driving laboratory
• vision-language model
• robotic benchmark
• fine-grained evaluation