field testing

To address the “simulation-to-reality gap”—the difficulty of reproducing simulation-identified failure scenarios in real-world autonomous driving—this paper proposes a verification method based on formal scenario modeling and time-series matching. The method formally translates abstract scenario programs written in the Scenic probabilistic programming language into computable temporal matching rules, enabling precise retrieval of failure-relevant patterns from large-scale real-world sensor data. A key contribution is the design of an efficient, linearly scalable query algorithm that supports real-time pattern matching over long temporal sequences. Experimental evaluation demonstrates that the approach achieves higher recall accuracy for critical failure scenarios than state-of-the-art commercial vision-language models, while accelerating query throughput by several orders of magnitude. This significantly improves both the efficiency and trustworthiness of transferring simulation-discovered failures to real-vehicle validation.

In cloud-native microservices, manual and fragmented observability configuration leads to slow fault localization, high resource overhead, and degraded system performance. This paper introduces the first continuous observability assurance methodology, shifting from experience-driven to experiment-driven design. Built upon the Observability eXperimentation (OXN) framework, our approach integrates A/B testing, metric-based feedback loops, and Infrastructure-as-Code (IaC)-enabled automation to dynamically optimize and quantitatively evaluate observability configurations. Evaluated in realistic microservice deployments, our method reduces mean time to detection by 42% on average, decreases sampling overhead by 31%, and—uniquely—enables quantitative validation of how specific observability configurations directly impact Service-Level Objective (SLO) compliance. By establishing a reproducible, iterative, and empirically grounded design paradigm, this work advances observability engineering from ad hoc practice to rigorous, data-driven discipline.

Current evaluations of large language model (LLM) agents for database administration lack production-level fidelity and suffer from inconsistent tasks, platforms, and metrics, hindering meaningful cross-study comparisons. To address this, this work proposes DBA-Bench, a high-fidelity benchmark for database operations that introduces, for the first time, production-grade realism, outcome-oriented evaluation, and controllable scenario replay, covering four critical real-world operational gaps. Built on PostgreSQL instances under active workload, DBA-Bench integrates multi-source time-series and log observations, snapshot rollback, deep annotations, and an automated evaluation framework, encompassing 106 scenarios across seven task categories. In 848 automated runs, the best-performing agent achieved a safety pass rate of only 17.9%, starkly lower than human DBAs’ 93.4%, exposing significant limitations of current approaches in complex, safety-critical repair tasks.

Existing search-based software testing (SBST) methods for Simulink models struggle to directly support natural-language Requirements Tables (RTs), necessitating cumbersome manual formalization of requirements into logical constraints. Method: This paper proposes the first black-box SBST framework natively driven by RTs. It introduces a semantic parsing and constraint mapping mechanism that automatically translates natural-language requirements in RTs into executable test constraints, integrated with genetic algorithms and Simulink’s simulation interface for end-to-end automated test generation—bypassing explicit logical formula translation. Contribution/Results: Evaluated on 60 real-world model–RT pairs, the approach achieves a 70% failure-revealing test case generation rate, matching the efficiency of state-of-the-art non-RT-based SBST tools. Moreover, it uncovered three critical failures in a cruise control model missed by other tools, demonstrating both industrial applicability and technical novelty.

In cloud-native systems, alert rules frequently suffer from false positives and false negatives due to the absence of design-phase validation, while existing tools lack systematic support for alert testing. To address this, we propose the “Alert-as-Experiment” paradigm—the first adaptation of the observability experimentation framework OXN to early-stage alert rule validation. Our approach enables closed-loop, development-time testing and continuous calibration of alert logic via simulated execution, synthetic observation data injection, and real-world scenario replay. It supports parameter tuning and repeatable verification of alert-triggering behavior, shifting alert engineering from empirical practice toward a testable, verifiable, and systematic discipline. Empirical evaluation demonstrates significant reductions in both false positive and false negative rates in production environments, alongside improved fault response latency and system maintainability.

Existing agent evaluation benchmarks predominantly focus on virtual software interactions and fail to assess the multimodal interface coordination and feedback-driven parameter tuning required for scientific instrument control. This work introduces the first benchmark specifically designed for this domain, presenting a web-based, extensible, secure, and reproducible simulator suite encompassing eight instrument types and 96 subtasks that fully span the workflow from sample loading to result inspection. The benchmark supports flexible task configuration and execution-based evaluation, integrating vision-language models with a dedicated agent framework. Experimental results demonstrate that while current agents can handle structured GUI subtasks, they struggle significantly with feedback-driven operations and long-horizon workflows, thereby validating the benchmark’s necessity and its capacity to expose critical gaps in agent capabilities.

This study addresses the challenge of effectively validating input model specifications in digital twin simulations, where conventional approaches—relying solely on marginal output distributions—often fail to detect misspecified joint input models. To overcome this limitation, the authors propose a novel statistical validation framework based on sub-trajectory conditioning. By repeatedly restarting simulations from observed system states while conditioning on subsets of random inputs, the method constructs conditional output distributions that enable goodness-of-fit testing of the full joint input model. This approach innovatively transcends the constraints of marginal validation and is complemented by diagnostic tools to pinpoint specific input sources responsible for detected discrepancies. Empirical evaluations on M/M/1 and tandem queueing systems demonstrate the framework’s heightened sensitivity and effectiveness, successfully identifying input model misspecifications that traditional methods overlook.

This work addresses the challenge of reliably handling irreversible, safety-critical experimental failures in chemical self-driving laboratories, where robots lack fine-grained failure data and standardized evaluation protocols. The study introduces the first fine-grained (11-class) robotic failure analysis framework tailored for chemical experiments, comprising a simulation environment with controllable multi-level (control, physical, semantic) fault injection, a large-scale multidimensional failure trajectory dataset, a six-dimensional capability evaluation benchmark, and a dedicated vision-language diagnostic model. A structured diagnosis and recovery instruction generation mechanism enables closed-loop fault tolerance. The fine-tuned vision-language model achieves 92.58% failure detection accuracy and 85.58% temporal localization accuracy in seen environments, and when deployed as a real-time supervisor, it improves downstream task success rates by 10–20 percentage points.

Traditional benchmarks provide only aggregate scores, offering insufficient evidence to support reliable deployment decisions and thereby creating a disconnect between evaluation and action. To address this gap, this work proposes a “deployment-completeness” benchmarking framework, introducing novel metrics—evidence fibers, completeness curves, and certifiable proportions—alongside a systematic audit methodology comprising evidence fiber analysis, response ranking intervals, conformal coverage evaluation, and a certify-then-acquire decision pipeline. Empirical evaluation on benchmarks such as Tox21, Matbench, and JARVIS reveals that conventional approaches suffer a drastic drop in channel coverage to 10.07% under real-world deployment conditions. In contrast, the proposed method reduces error-driven deployment decisions to 0.027% on Tox21 and 0.128% on JARVIS, substantially enhancing deployment reliability.

Existing SRE benchmark tasks are overly simplified and fail to capture the complexity of fault diagnosis and mitigation in real-world production environments. This work proposes the first high-fidelity, scalable evaluation benchmark for SRE agents, built upon a realistic cloud-native system stack that dynamically simulates operational conditions. The benchmark incorporates a fault injector and noise simulator to support diverse failure modes—including metastable and correlated failures—and provides 90 realistic, challenging tasks. Designed with a modular architecture, it enables continuous extension and adaptation. Experimental results demonstrate significant performance disparities among state-of-the-art AI agents across different fault types, with end-to-end success rates varying by up to 40%, thereby validating the benchmark’s effectiveness and inherent difficulty.