A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

2026-06-08

AI Summary

Current evaluations of vision-language-action (VLA) policies suffer from a lack of reliable correlation between simulation performance and real-world outcomes, limiting their utility in guiding practical deployment. This work presents the first systematic quantification of sim-to-real alignment across multiple simulation platforms, analyzing consistency in policy ranking, performance correlation, and failure modes under perturbations across diverse tasks. Through extensive multi-platform simulations, policy fine-tuning analyses, post-training data scaling studies, and robustness evaluations, the study identifies key characteristics of high-fidelity simulators and proposes design principles to enhance simulation utility. These findings substantially improve the reliability and practical guidance value of simulation in VLA policy development.

Abstract

Simulation has become an essential tool for evaluating and improving vision-language-action (VLA) policies, offering scalable, reproducible, and controllable alternatives to costly real-world robot evaluation. Recent simulation benchmarks have made substantial progress on realism and diversity, yet these platforms have not been widely adopted as reliable proxies for real-world policy evaluation. In this work, we investigate this issue through the lens of sim-and-real correlation. We conduct a systematic study across multiple simulation platforms, VLA policies, tasks, and perturbation factors, measuring whether simulated evaluation preserves real-world conclusions in terms of policy ranking consistency, performance correlation, and perturbation-wise failure patterns. This analysis allows us to characterize the limitations of existing simulators and identify what kinds of simulation signals are more aligned with real-world deployment. We further examine how users should exploit simulation for policy improvement, including when simulator-based finetuning is beneficial and how the amount of post-training data affects sim-and-real alignment. Overall, our work provides a unified framework for measuring, interpreting, and improving the usefulness of simulation for VLA policies, offering guidance both for simulator designers and for practitioners who use simulation as part of the policy development pipeline.

Problem

• sim-to-real
• vision-language-action
• simulation evaluation
• policy correlation
• robotics

Innovation

• sim-to-real correlation
• vision-language-action (VLA)
• simulation benchmarking
• policy evaluation
• robotic simulation