android instrumentation

This study addresses the high maintenance costs and poor stability of UI testing in large-scale enterprise applications by proposing the first multi-agent autonomous testing system empirically evaluated in a real-world enterprise environment. The system integrates large language models, LangGraph orchestration, the Playwright execution engine, and a RAG-augmented knowledge base, enhanced with runtime DOM analysis to enable goal-free functional discovery, dynamic coverage expansion, and self-repair of failing tests. Experimental results across ten UI scenarios demonstrate that the system discovers over 100 testable functionalities, achieves a 70% scenario-level repair convergence rate (averaging 3.4 iterative rounds), yet only 10% of scenarios succeed on the first attempt and 38% yield ineffective executions—highlighting how unconstrained autonomy can lead to semantic drift. The findings underscore constrained autonomy and human-in-the-loop collaboration as essential pathways toward reliable autonomous testing.

Web applications exhibit high interface dynamism and complex navigation and form interactions, posing significant challenges for end-to-end test case generation—particularly low coverage and poor robustness. To address this, we propose a navigation modeling approach that integrates screen transition graphs with large language models (LLMs) to enable high-precision path exploration. We further design a state-graph-based automated testing framework for conditional forms, supporting dynamic DOM parsing and interaction modeling. Additionally, we construct the first dedicated benchmark dataset for form-interaction testing. Experimental evaluation across diverse, complex web applications demonstrates substantial improvements in test coverage and path discovery accuracy—especially for branching navigation and conditional form-filling tasks. Our approach establishes a scalable, empirically evaluable paradigm for web application reliability testing.

This study addresses the imbalance in the test pyramid—characterized by an overreliance on coarse-grained integration and system tests, which leads to difficulties in fault localization and slow execution—by proposing, for the first time, a method to automatically generate unit tests from existing integration tests. The approach combines static and dynamic analysis to automatically isolate component dependencies and enhance coverage at the unit level. Implemented as a Node.js tool and evaluated on twelve open-source JavaScript projects, the technique produces high-quality unit tests that significantly improve test suite structure, thereby increasing both testing efficiency and maintainability.

Existing pixel-level and region-level approaches for UI visual regression testing struggle to model semantic context, spatial relationships, and subtle visual changes. To address this, we propose a graph neural network (GNN)-driven, context-aware visual change detection method. Our core contribution is the first explicit modeling of UI interfaces as structured graphs: nodes represent detected UI controls, while edges encode their spatial and semantic relationships. Leveraging graph alignment and multi-scale GNN inference, our method achieves precise cross-version control matching and fine-grained change localization. Experimental evaluation across diverse, complex UI scenarios demonstrates over 35% improvement in accuracy compared to conventional baselines. The approach has been successfully integrated into real-world software iterative testing pipelines, effectively overcoming the semantic understanding limitation inherent in purely pixel-based visual comparison techniques.

This study addresses the challenges of unstable end-to-end testing for Android applications in continuous integration (CI) due to fragile emulator configurations. It presents the first large-scale empirical analysis of 4,518 open-source projects, systematically examining how instrumentation tests are configured, how these practices evolve, and their comparative effectiveness in CI environments. Leveraging GitHub Actions metadata, the work evaluates three prevalent approaches: Gradle Managed Devices, community-reusable components, and custom scripts. Findings reveal that only 10.6% of projects adopt such testing; among them, community components demonstrate superior reliability and efficiency, third-party device labs are suitable for regression testing despite higher costs, and custom scripts, while flexible, suffer from high retry rates. The study thus illuminates current practices and critical trade-offs in Android CI testing.

Current verification workflows for autonomous systems suffer from a lack of coordination among scenario design, simulation execution, and telemetry analysis, leading to poor traceability between requirements, tests, and evidence, which undermines reproducibility and debugging efficiency. This work proposes a unified verification framework powered by large language models (LLMs) that bridges this gap through task-level structured scenario representations. The framework automatically translates high-level verification intents into temporally evolving scenarios, enabling automated simulation execution and context-aligned telemetry analysis. Furthermore, it incorporates a counterfactual scenario generation mechanism driven by failure cases to establish a closed-loop, self-evolving testing process. The approach substantially enhances traceability, reproducibility, and scalability of verification, accelerates test iteration cycles, and deepens insight into system behavior.

This work addresses the limited accessibility of large language model (LLM) and agent workflow development for engineers without machine learning expertise, primarily due to the absence of integrated testing, debugging, and reproducibility capabilities. To bridge this gap, the authors propose a novel IDE-native AI observability workflow, implemented as the AI Toolkit plugin for JetBrains IDEs. This approach seamlessly embeds trace capture and evaluation into standard run/debug cycles, enabling automatic hierarchical trace logging during execution, one-click dataset persistence, and a pluggable, unit-test-like evaluation framework. By minimizing environment setup and context-switching overhead, the solution facilitates routine evaluation and immediate trace visualization. Empirical data from the initial PyCharm release demonstrates high adoption, sustained usage, and low churn, confirming that IDE-integrated tooling effectively lowers the barrier to entry for non-ML developers.

It remains unclear how exploration strategies and state abstraction jointly influence the effectiveness of automated Web GUI testing. This study presents the first systematic evaluation of the combined impact of three prominent exploration strategies—model-based, reinforcement learning, and large language model–driven approaches—with various state abstraction mechanisms on code coverage and bug detection. The experiments encompass six state abstractions and multiple history representations, revealing that no single strategy universally dominates: fine-grained abstractions better suit model-based methods, compact representations benefit reinforcement learning, and functional-level context most effectively enhances large language model performance. Furthermore, the work demonstrates only a weak correlation between code coverage and bug discovery, underscoring the necessity of evaluating both metrics complementarily.

This study addresses the challenges in testing Internet of Things (IoT) software, particularly the complexity of external dependencies and insufficient test case effectiveness, which have lacked systematic empirical investigation. It presents the first large-scale analysis of testing practices in open-source IoT projects, integrating assessments of test effectiveness, categorization of testing challenges, and mining of mock usage patterns. The findings reveal that despite the substantial volume of tests, their effectiveness is generally limited, with managing external dependencies emerging as a central difficulty. Moreover, the judicious application of mock objects significantly enhances test coverage and quality. This work establishes the first empirical benchmark for IoT software testing and offers concrete directions for improving testing practices in this domain.