robot experiment design

This work proposes a novel “betting”-based methodology for efficiently and accurately evaluating robotic performance in real-world settings where physical experimentation is constrained. By introducing betting theory into sim-to-real performance assessment—a first in the field—the approach constructs an estimator theoretically superior to Monte Carlo estimation. The method integrates control variate approximation, cross-fidelity simulation, and statistical decision rules to enable practical deployment. Experimental results demonstrate its efficacy on synthetic data and simulated environments, and it successfully infers real-world robotic grasping accuracy with significantly fewer physical trials while improving evaluation precision.

This study addresses the persistent gap between theoretical control performance and its practical realization in real-world robotic systems, often caused by inadequate discretization, insufficient real-time guarantees, and weak error handling in control software. For the first time from a software engineering perspective, the authors systematically analyze 184 open-source robotic controllers through code review, empirical analysis, and test evaluation, uncovering common deficiencies in application scenarios, implementation details, and verification practices. The findings reveal that most implementations fail to properly account for critical system constraints, and their testing strategies inadequately validate the theoretical assurances they claim. This work highlights a significant disconnect between implementation quality and theoretical promises, offering concrete directions and practical guidelines for developing reliable, verifiable robotic control software.

This work addresses the non-reproducibility of ROS 2 simulations across different hardware platforms, a critical issue stemming from its asynchronous multi-process architecture that undermines scientific evaluation and continuous integration. We propose a non-intrusive approach that enforces deterministic execution by unifying the callback scheduling order of ROS 2 C++ nodes without requiring any modifications to existing codebases. To our knowledge, this is the first method to achieve fully reproducible simulation for ROS 2 C++ nodes. We introduce the ROS Simulation Library for C++ (RSLCPP), which provides precise control over execution timing and communication sequencing. Experimental validation on both synthetic benchmarks and real-world robotic systems demonstrates that our approach yields bit-for-bit identical simulation results across diverse CPU architectures. The implementation is publicly available as open-source software.

How can the openness, reproducibility, and trustworthiness of autonomous robotic scientific experiments be ensured? This paper proposes a semantic execution tracing framework built upon the AICOR Virtual Research Building platform, which semantically aligns robotic belief states with heterogeneous sensor data. By integrating digital twin technology, a deterministic execution engine, and a cloud-native architecture, the framework enables end-to-end traceable logging and cross-platform experimental reproducibility. Its key innovation is the first implementation of a robot experiment digital twin supporting semantic memory and real-time verification—rendering experimental processes transparent, reasoning auditable, and results independently verifiable. This paradigm significantly enhances the reliability and collaborative efficiency of automated scientific research, providing foundational infrastructure for autonomous systems to meaningfully contribute to scientific discovery. (149 words)

Real-world validation of learned robotic systems incurs high costs and suffers from scarce empirical data, making it difficult to establish high-confidence performance guarantees. To address this, we propose a variance-reduction estimation framework leveraging cross-platform paired data—specifically, introducing the control variates method into robotic performance evaluation for the first time. By exploiting strong correlations between simulated and real-world observations, our approach constructs auxiliary estimators that, when combined with Monte Carlo estimation, enable theoretically controllable variance reduction. We validate the method on autonomous driving and quadrupedal locomotion tasks, demonstrating over 50% reduction in required real-world samples at equivalent confidence levels—significantly improving sample efficiency and lowering validation costs. Our core contribution is the systematic establishment of a control-variates-based evaluation paradigm tailored to learning-based robotic systems, uniquely balancing theoretical rigor with practical engineering applicability.

This study addresses the limitations of current robotic system validation, which relies heavily on manual selection of test scenarios, thereby hindering scalability and compromising reproducibility and reliability of conclusions. To overcome these challenges, this work proposes a compositional, scenario-based modeling approach that integrates declarative test specifications, plugin-driven scenario generation, containerized parallel simulation, and unified result analysis to establish the first modular and scalable automated verification framework. The framework enables systematic parameter variation across multiple dimensions and facilitates robust identification of systemic faults versus stochastic anomalies. Evaluated across 5,480 distinct scenario configurations with over 100,000 simulation runs, the approach accumulated 1,800 hours of simulated operation and 1,873 virtual kilometers, demonstrating its efficacy in discerning consistent system deficiencies from random irregularities.

This work addresses the challenge of efficiently and reliably inferring the mean performance of robotic systems in real-world environments from simulation data. It introduces, for the first time, a novel integration of sim-to-real performance estimation with betting-based E-processes to construct anytime-valid confidence sequences. By combining a scaled simulator with an anytime-valid inference framework, the method dynamically generates statistically rigorous confidence bounds on true performance. Experimental results on synthetic data demonstrate that the proposed approach efficiently produces accurate confidence certificates for mean performance estimates, thereby offering a principled statistical guarantee for sim-to-real performance evaluation.

This work investigates the sim-to-real transferability of vision-language-action (VLA) models to a physical UR5e robotic arm and presents an end-to-end pipeline encompassing real-world data collection, construction of an RLDS-compatible dataset, fine-tuning of OpenVLA models, and their deployment. By incorporating multimodal temporal alignment, unified coordinate systems, and consistent action semantics, the study demonstrates that successful real-world performance hinges on holistic co-design across the data–model–control pipeline rather than model performance alone. A reproducible evaluation framework reveals a substantial gap between offline metrics and closed-loop task success, primarily attributable to system-level factors such as action semantics, image preprocessing, and data quality.

This work addresses the challenges of scaling robotic education in higher education, where high costs of commercial digital twins and the steep learning curve of open-source middleware like ROS hinder widespread adoption. To overcome these barriers, the authors propose an education-oriented, four-layer extensible communication architecture that abstracts complex communication protocols and enables seamless integration between graphical modeling environments and physical robots. Notably, this architecture is the first in an open-source platform to support hardware-agnostic, high-fidelity virtual–physical mapping. By integrating 3D visual modeling, a ROS backend, and efficient data serialization and routing mechanisms, the system significantly reduces deployment complexity. Experimental validation through multi-axis spatial trajectory tasks demonstrates its effectiveness in facilitating practical robotics instruction in engineering education.

Evaluating general-purpose robotic policies in the real world is hindered by the combinatorial explosion of task factors, high testing costs, and insufficient coverage of existing test sets, making it difficult to comprehensively assess deployment reliability. This work formulates evaluation as a sequential experimental design problem and introduces the first active, factorized evaluation framework for real-world robot policies. By leveraging Bayesian optimization and probabilistic surrogate models, the approach adaptively selects test configurations with maximal information gain within a structured task factor space, efficiently characterizing policy performance under unseen conditions. Across three tasks and 2,331 real-world trials, the method reduces the number of required tests by 20–40% compared to random testing, significantly improving evaluation efficiency and coverage while systematically identifying failure regions.