interaction logging

This study addresses the challenge of effectively monitoring early-stage agent systems, where structural flaws often obscure task-level errors. The authors propose a three-dimensional (quality, suitability, efficiency) and three-granularity (intra-run, inter-run, structural) monitoring and triaging framework tailored for low-maturity agent systems. They introduce a novel system maturity staging model based on the coefficient of variation and monitoring granularity, integrated with a severity classification adapted from FMEA to guide human review. The resulting transferable monitoring architecture supports document-driven, multi-stage workflows, enhanced by a synthetic testbed with controlled error injection. Experimental results demonstrate that structural defects significantly mask task-level signals; 97% of issues can be automatically traced, with only 2% requiring human intervention, and each granularity level precisely identifies its corresponding defect type (coefficients of variation: 0.02, 1.25, and 0.00, respectively).

To address the “simulation-to-reality gap”—the difficulty of reproducing simulation-identified failure scenarios in real-world autonomous driving—this paper proposes a verification method based on formal scenario modeling and time-series matching. The method formally translates abstract scenario programs written in the Scenic probabilistic programming language into computable temporal matching rules, enabling precise retrieval of failure-relevant patterns from large-scale real-world sensor data. A key contribution is the design of an efficient, linearly scalable query algorithm that supports real-time pattern matching over long temporal sequences. Experimental evaluation demonstrates that the approach achieves higher recall accuracy for critical failure scenarios than state-of-the-art commercial vision-language models, while accelerating query throughput by several orders of magnitude. This significantly improves both the efficiency and trustworthiness of transferring simulation-discovered failures to real-vehicle validation.

Safety-critical small Unmanned Aircraft Systems (sUAS) lack systematic, standardized testing processes that are tightly integrated with safety analysis. Method: This paper proposes a requirement-driven coupled testing framework, introducing the novel triadic paradigm of “requirements–simulation testing–safety analysis.” It employs formal requirement modeling with bidirectional traceability, a simulation–hardware-in-the-loop cooperative testing architecture, scenario-driven test case generation, and deep integration of safety analysis methods (e.g., Fault Tree Analysis and System-Theoretic Process Analysis). Contribution/Results: Evaluated on an sUAS case study, the framework significantly improves simulation fidelity coverage and requirement coverage, enables end-to-end safety evidence generation, fills the gap in standardized sUAS testing procedures, and delivers reproducible, verifiable testing assets to support airworthiness certification.

Spectrum-Based Fault Localization (SBFL) fails when no failing tests are available to trigger faults. Method: This paper systematically demonstrates, for the first time, that stack traces from crash reports can serve as pseudo-failure signals in lieu of actual failing tests, and proposes SBEST—a novel SBFL method that integrates exception-location semantics with method-call-graph reachability to embed stack-trace information into the spectrum analysis framework. SBEST jointly leverages test coverage matrices and parsed stack traces to enable precise fault localization even in the absence of failing tests. Results: Experiments show SBEST improves Mean Average Precision (MAP) by 32.22% and Mean Reciprocal Rank (MRR) by 17.43% over the baseline MAP method. Moreover, 98.3% of defect-fixing intentions align with stack-trace anomalies, and 78.3% of defective methods are reachable within an average of 0.34 call-graph hops. This work establishes a new lightweight, crash-driven paradigm for fault localization.

This study addresses the challenges in testing Internet of Things (IoT) software, particularly the complexity of external dependencies and insufficient test case effectiveness, which have lacked systematic empirical investigation. It presents the first large-scale analysis of testing practices in open-source IoT projects, integrating assessments of test effectiveness, categorization of testing challenges, and mining of mock usage patterns. The findings reveal that despite the substantial volume of tests, their effectiveness is generally limited, with managing external dependencies emerging as a central difficulty. Moreover, the judicious application of mock objects significantly enhances test coverage and quality. This work establishes the first empirical benchmark for IoT software testing and offers concrete directions for improving testing practices in this domain.

This work addresses the significant challenges of crash diagnosis in ultra-large-scale industrial mobile applications operating in mixed-language environments, where code complexity and non-reproducible failures hinder effective debugging. The authors propose Holmes, a novel multi-agent system that automatically reconstructs failure contexts and identifies root causes without requiring environment reproduction by innovatively integrating multimodal runtime signals—such as stack traces, logs, and thread states. Holmes employs a hierarchical retrieve-explore-reason architecture, leveraging low-level information like registers and assembly instructions to bridge the semantic gap between open-source application logic and closed-source system frameworks. Guided by runtime clues, it dynamically narrows the search space through intelligent code navigation. Evaluated on real-world crash data from WeChat, Holmes achieves an 87.6% function-level localization accuracy and reduces average diagnosis time by 98%, down to approximately 77 seconds.

This study addresses the challenges posed by rapid evolution in digital forensic systems and tools, which induces drift in evidentiary behaviors and tool outputs, thereby undermining result reproducibility and trustworthiness. To mitigate this, the authors propose a test-driven forensic methodology that introduces state-transition testing for causal attribution, encoding forensic expectations as executable specifications. The approach integrates virtual machine environments with computer vision–guided GUI automation to simulate authentic user interactions and verify system state changes. An open web platform is developed to facilitate sharing and replication of experiments. The method’s efficacy is demonstrated through five case studies, including a regression analysis across 25 versions of Autopsy, which uncovered numerous undocumented, substantial changes in its reporting output.

This study addresses the critical challenge of network outages caused by misconfigurations in large-scale networks, where existing large language models (LLMs) exhibit limited repair capabilities and often introduce new errors. To overcome these limitations, this work proposes an agent-based architecture that integrates formal verification with contextual retrieval to dynamically manage configuration context and iteratively validate repair proposals. As the first systematic evaluation of LLM agents for network configuration repair, the experimental results demonstrate that the proposed approach significantly outperforms baseline methods on both open-source and closed-source LLMs, achieving a 12% average improvement in repair success rate and a 17% increase in safety, effectively resolving original misconfigurations while preventing the introduction of new faults.

Existing agent evaluation benchmarks predominantly focus on virtual software interactions and fail to assess the multimodal interface coordination and feedback-driven parameter tuning required for scientific instrument control. This work introduces the first benchmark specifically designed for this domain, presenting a web-based, extensible, secure, and reproducible simulator suite encompassing eight instrument types and 96 subtasks that fully span the workflow from sample loading to result inspection. The benchmark supports flexible task configuration and execution-based evaluation, integrating vision-language models with a dedicated agent framework. Experimental results demonstrate that while current agents can handle structured GUI subtasks, they struggle significantly with feedback-driven operations and long-horizon workflows, thereby validating the benchmark’s necessity and its capacity to expose critical gaps in agent capabilities.