Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis

2026-06-07

AI Summary

This work addresses the lack of systematic evaluation of robustness in vision-language-action (VLA) models on low-cost real-world robots, particularly the absence of in-depth analysis of failure causes and recovery capabilities. The authors establish the first standardized real-world benchmark for VLA models on the affordable SO-101 platform, covering four representative manipulation tasks and employing a unified evaluation protocol with real teleoperation data to fine-tune multiple policies—including π₀.₅, SmolVLA, Wall-X, and ACT. They introduce a novel structured failure taxonomy that integrates semantic- and execution-level attributes, along with recovery-aware evaluation metrics. Experiments reveal that pretrained VLA policies generally outperform imitation learning baselines but exhibit strong task dependency; execution instability emerges as the primary failure mode, and recovery capabilities vary significantly across architectures, underscoring the necessity of fine-grained failure and recovery analysis.

Abstract

Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted in simulation or on expensive robotic platforms, leaving their robustness on affordable real-world robots largely unexplored. We present a standardized real-world benchmark for evaluating representative VLA and imitation learning policies on the low-cost SO-101 robotic platform. The benchmark comprises four representative manipulation tasks together with unified evaluation protocols, enabling systematic comparison under embodiment uncertainty. Using real-world teleoperated demonstrations, we fine-tune and evaluate $π_{0.5}$, SmolVLA, Wall-X, and ACT directly on the physical platform. Beyond conventional task success rates, the benchmark incorporates a structured failure taxonomy, semantic- and execution-level failure decomposition, and recovery-aware evaluation metrics to characterize policy robustness. Experimental results show that stronger pretrained VLA policies generally outperform the imitation learning baseline, although performance remains highly task-dependent under low-cost robotic deployment conditions. Execution instability emerges as the dominant failure source, while recovery capability varies substantially across architectures. These results highlight the importance of failure and recovery analysis beyond binary task success and establish SO-101 as a practical benchmark for evaluating embodied AI systems under realistic low-cost robotic deployment conditions.

Problem

• Vision-Language-Action
• robotic manipulation
• failure analysis
• recovery
• low-cost robotics

Innovation

• Vision-Language-Action models
• real-world benchmarking
• failure analysis
• recovery-aware evaluation
• low-cost robotics