automated testing

Despite growing adoption of AI-driven test automation tools, their real-world efficacy—particularly in improving test efficiency, reducing maintenance costs, and enhancing defect detection—remains inadequately evaluated. Method: We conduct a systematic literature review identifying 55 tools and propose the first taxonomy of AI testing capabilities; further, we perform a dual-tool, dual-system empirical study on open-source projects, evaluating core functionalities including UI self-healing, visual testing, and intelligent test case generation. Contribution/Results: AI tools improve execution efficiency and reduce maintenance effort by over 30%, yet suffer from high false-positive rates, insufficient domain knowledge integration, and strong model dependency. This work establishes the first benchmarking framework for AI-based testing that jointly integrates a comprehensive capability taxonomy with multi-dimensional empirical validation—providing foundational guidance for developing robust, interpretable, and production-ready AI testing tools.

Traditional regression testing theory fails in agile and continuous integration settings due to the dynamic, time-ordered nature of continuous builds. Method: This paper proposes a formal modeling framework based on time-ordered build chains, representing continuous build sequences as temporally constrained build-tuple chains and formally defining the novel concept of the “regression testing window”—unifying classical two-version and multi-version continuous testing scenarios. The framework enables efficient verification of correctness and completeness of regression testing within bounded time and supports rigorous formal verification via logical deduction. Contribution/Results: Experimental evaluation demonstrates that the model successfully characterizes and verifies two state-of-the-art agile regression testing algorithms. It exhibits strong expressive power, requires no auxiliary assumptions, ensures theoretical soundness, and is directly deployable in practice.

This study addresses the challenges of regression testing in remote and hybrid work environments, where communication, coordination, and quality assurance are increasingly complex. Through qualitative interviews with 20 software practitioners, complemented by process analysis, tool integration assessment, and coding of collaborative practices, the research systematically investigates the sociotechnical evolution of regression testing in distributed settings. Findings indicate that while core testing phases remain largely stable, teams increasingly rely on documentation, automation, and integrated toolchains to sustain effectiveness. Standardized reporting formats, shared repositories, and traceability mechanisms significantly mitigate collaboration barriers inherent in remote work. The study offers novel insights and practical guidance for ensuring software quality in geographically dispersed development contexts.

In enterprise microservice regression testing, QA engineers often lack up-to-date documentation and must rely on real user traffic to reconstruct business scenarios; however, transforming such traffic into replayable test cases with stable assertions is labor-intensive and error-prone. This work proposes NL2Test, a method that combines the semantic understanding of large language models with deterministic algorithms to generate executable API test cases end-to-end from natural language scenario descriptions and captured execution traces. NL2Test automatically slices request sequences, reconstructs data dependencies, masks non-deterministic fields, and produces business-aligned, reliable assertions. Evaluated on 51 industrial scenarios, it achieves an exact match rate of 82.4%, with 98.0% of generated test cases becoming functional after minor tuning. During a nine-month production deployment, it produced 3,196 test cases, of which 85.4% were accepted and integrated into the codebase.

This work addresses the bottleneck in agile development where the rapid generation of test specifications far outpaces their manual conversion into executable scripts, leading to backlogs and delayed releases. To bridge this gap, the authors propose an embodied AI collaborator architecture tailored for industrial settings, featuring a retrieval-augmented multi-agent system that enables end-to-end generation of system-level automated test scripts from validated specifications. The framework seamlessly integrates into existing agile workflows while preserving human oversight through mandatory review gates. Evaluated in a real-world deployment at Hacon, a Siemens subsidiary, the approach significantly reduces the manual scripting burden and demonstrates that clear specification standards combined with effective human-AI collaboration are pivotal for scaling regression test automation.

Traditional software testing suffers from low coverage, high manual effort, and delayed feedback—leading to defect leakage and release delays. To address these challenges, we propose an AI-driven self-healing test case generation and validation framework. Our approach introduces a novel dynamic testing mechanism that integrates risk-aware machine learning models with explainability enhancements (LIME/SHAP), enabling end-to-end generation of test cases from natural language requirements, identification of risk hotspots, real-time test prioritization, and code-change-driven continuous regression adaptation. The framework balances automation depth with human-in-the-loop controllability. Empirical evaluation across multiple industrial systems demonstrates a 37% improvement in test coverage, a 52% reduction in regression cycle time, and a 29% increase in defect detection rate. Moreover, it maintains full compatibility with legacy systems and cloud-native architectures.

This study addresses the imbalance in the test pyramid—characterized by an overreliance on coarse-grained integration and system tests, which leads to difficulties in fault localization and slow execution—by proposing, for the first time, a method to automatically generate unit tests from existing integration tests. The approach combines static and dynamic analysis to automatically isolate component dependencies and enhance coverage at the unit level. Implemented as a Node.js tool and evaluated on twelve open-source JavaScript projects, the technique produces high-quality unit tests that significantly improve test suite structure, thereby increasing both testing efficiency and maintainability.

Existing LLM-based automated test generation primarily produces static input-output assertion pairs, resulting in limited test diversity and insufficient debugging information. This work proposes a novel paradigm for generating executable test harnesses—supporting dynamic input construction and flexible output validation (e.g., invariant checking). Methodologically, we design a two-stage training framework: first, supervised fine-tuning (SFT) to teach LLMs the structural conventions of test scripts; second, reinforcement learning with a custom reward function (RLVR) to optimize test quality along dimensions such as correctness, coverage, and verifiability. Empirical evaluation demonstrates substantial improvements in defect detection rate and test strategy diversity; moreover, the generated harnesses support runtime extension to further enhance code generation fidelity. Our core contribution is the first systematic advancement of LLM-driven test generation—from static assertion pairs to fully executable, formally verifiable, and extensible test programs.

This work addresses the challenge that rapid software development often compromises code maintainability, thereby hindering safe AI-assisted refactoring. To mitigate this, the authors propose an iterative refactoring approach that integrates large language models with human oversight. The method first leverages a code-specialized large language model to automatically generate high-coverage unit tests that capture existing program behavior. Subsequently, developers guide test-driven refactoring, while branch coverage metrics are used to constrain and validate model-generated outputs. Empirical evaluation demonstrates that the approach produces nearly 16,000 lines of reliable test code within hours, achieving up to 78% branch coverage on critical modules. This significantly reduces regression risk during large-scale refactoring and enhances the reliability and practicality of AI-assisted code restructuring.

This study addresses a critical yet previously underexplored issue in large language model (LLM)-driven software development: the contamination of automatically generated tests by erroneous code. The authors systematically uncover and empirically validate this error propagation phenomenon, demonstrating that when tests are generated based on incorrect code within multi-step agent workflows—across diverse programming tasks and various prompting strategies, including chain-of-thought—the resulting tests exhibit significantly lower defect detection rates (14%) compared to independently generated tests (25%). These findings challenge the prevailing assumption that LLM-generated tests can serve as reliable, independent oracles, thereby highlighting the substantial risk of test bias in LLM-augmented development pipelines.

This work addresses a critical limitation in existing benchmark suites such as SWE-bench, which often misclassify semantically incorrect yet superficially passing patches as valid, thereby overestimating the performance of automated program repair systems. To mitigate this issue, the authors propose STING, a novel framework that integrates semantic mutation with behavior-preserving transformations to establish a closed-loop test augmentation pipeline. By employing semantically mutated programs as diagnostic stressors, STING identifies inadequacies in existing test suites and generates targeted, high-fidelity test cases. Evaluated on SWE-bench Verified, STING reveals vulnerabilities in 77% of the instances, contributes 1,014 new effective tests, and improves line and branch coverage by 10.8% and 9.5%, respectively. Furthermore, it reduces the success rate of top repair models by 4.2%–9.0%, effectively distinguishing genuinely correct patches from those that merely pass superficially.