mutation testing

Traditional test adequacy metrics, such as code coverage and mutation testing, focus on implementation details and struggle to capture discrepancies between expected and actual program behavior. This work proposes an automated approach that extracts method-level expected behaviors from natural language documentation and source code, then maps them to existing test cases, thereby formalizing and empirically evaluating “behavioral gaps”—a dimension of test adequacy independent of structural metrics. By integrating natural language processing, static analysis, and behavioral mapping techniques, our method identifies 20,729 behaviors across ten Java open-source libraries with 93.1% precision, revealing that 17.5% of expected behaviors remain entirely untested. Notably, these gaps persist even in methods exhibiting high code coverage or high mutation kill rates, exposing a systematic deficiency in current testing practices—including automatically generated tests—in validating intended program behavior.

Traditional mutation testing struggles to generate mutants that exhibit subtle semantic differences and closely resemble real-world programming faults, thereby limiting test effectiveness. This work proposes a novel approach that, for the first time, integrates large language model–driven round-trip translation between code and natural language intent into mutation testing. By leveraging translation discrepancies and controlled perturbations of intended behavior, the method generates high-quality mutants with nuanced semantic variations. Empirical evaluation on 40 real faulty methods demonstrates that the proposed technique—referred to as RTM—significantly improves fault detection rates using substantially smaller test suites: with only 4 and 30 test cases, RTM detects on average 4× and 1.7× more faults, respectively, than conventional approaches, confirming its efficiency and practicality.

This study addresses the prediction of “latent faults” in software version evolution—specifically, mutants that survive in the current version but are subsequently killed in a later version (termed “latent mutants”). We formally define this concept for the first time and empirically observe an average latency of 104 days, demonstrating both observability and timeliness. Methodologically, we generate mutants using PITest and develop a random forest model integrating mutant operator types with code change features. Evaluations across 13 open-source projects comprising 131,000 mutants show that latent mutants constitute 3.5% of all mutants, with our model achieving 86% accuracy and 67% balanced accuracy. Key contributions include: (1) formal definition and empirical characterization of latent mutants; (2) discovery of systematic correlations between latent mutants and both code evolution patterns and mutation operators; and (3) the first high-accuracy, interpretable predictive framework for latent fault identification.

Prior work lacks systematic empirical evaluation of large language models’ (LLMs) capability to generate high-quality mutants for mutation testing. Method: This paper presents the first large-scale empirical study, evaluating six open- and closed-source LLMs—including GPT-4 and CodeLlama—using multi-strategy prompt engineering on the Defects4J 2.0 and ConDefects Java benchmarks. Contribution/Results: LLM-generated mutants exhibit significantly higher behavioral similarity to real faults and achieve a 93% fault detection rate—19 percentage points higher than traditional rule-based approaches—along with markedly improved diversity. However, LLMs underperform conventional methods in compilation success rate and in producing fewer equivalent or non-viable mutants. This work establishes the first comprehensive empirical foundation and quality assessment framework for LLM-driven intelligent mutation testing.

Existing mutation testing relies on fixed operators (e.g., operator replacement, statement deletion), limiting its ability to emulate realistic faults and thereby constraining evaluation validity. This paper proposes the first large language model (LLM)-based dynamic mutant generation method: it identifies critical code locations, injects placeholders, and leverages prompt engineering to guide LLMs—including Codex, Llama, and GPT—to produce semantically coherent and high-fidelity mutants. Our approach breaks free from conventional operator constraints, enabling the first LLM-driven, context-aware, and dynamically constructed high-fidelity mutants. It captures real-world defect patterns inaccessible to tools like StrykerJS and supports multi-prompt strategies and cross-LLM adaptability. Evaluated on 13 open-source JavaScript packages, our method significantly improves mutant diversity and realism while maintaining practical execution efficiency and controllable computational cost.

Traditional mutation testing, which operates at the syntactic level, struggles to detect semantic defects arising from misunderstandings of program intent. This work proposes an intent-based mutation testing approach that, for the first time, treats programming intent itself as the unit of mutation. Leveraging large language models (LLMs), the method semantically rewrites natural language descriptions of intent and automatically generates executable program mutants. By shifting the focus from syntactic transformations to semantic reinterpretations of intent, this approach produces mutants that are both richer in semantics and more structurally complex. Empirical evaluation on 29 programs shows that 55% of the intent-based mutants are not covered by traditional mutation operators and exhibit significant differences from them in both syntactic and semantic dimensions, thereby substantially enhancing the detection of specification- and behavior-level faults.

This study addresses the lack of transparency in code-modification behaviors of current AI-powered coding agents during performance optimization. It presents the first empirical analysis of 1,254 code diffs from 216 AI-generated performance-optimization pull requests, systematically annotated using a dual-LLM cross-validation protocol based on an 18-category syntactic mutation taxonomy derived from Genetic Improvement (GI). The findings reveal a strong preference by AI agents for three mutation types: identifier renaming (37.0%), object creation (26.4%), and type changes (22.7%)—a stark contrast to traditional GI datasets, where over 84% of mutations involve no change. These results suggest that agent identity and optimization strategy can serve as effective priors for narrowing the search space of Search-Based Software Engineering (SBSE) operators.

This study addresses the lack of publicly available mutation testing benchmarks for IEC 61131-3 Structured Text (ST) programs widely used in industrial automation, which has hindered reproducible testing research. The authors present STMutants, the first mutation testing dataset specifically designed for PLC ST programs, incorporating seven mutation operators tailored to industrial control domains. Through a rigorous four-stage pipeline—syntactic transformation, compilation validation, manual equivalence screening (with inter-rater agreement κ = 0.87), and observability filtering—the dataset retains 108 high-quality, non-equivalent mutants. This benchmark facilitates research in automated test generation, fault localization, and AI-assisted quality assurance. Leveraging STMutants, the study evaluates three large language models, achieving mutation detection accuracies of 86.1%, 94.4%, and 86.1%, respectively, with statistical analysis confirming significant performance differences among them.

Existing research often relies on code coverage and mutation score as proxy metrics to evaluate the effectiveness of test cases generated by large language models (LLMs), yet their correlation with actual fault-detection capability remains unclear. This study conducts a large-scale empirical analysis of test suites produced by diverse LLMs across varied testing scenarios, systematically examining the relationships among coverage, mutation score, and real defect detection performance. The findings reveal that the validity of these proxy metrics is highly context-dependent: they offer some predictive value in regression testing but prove unreliable when the target code contains faults. Furthermore, the size of the test suite has limited influence on these correlations. These results challenge conventional assumptions in test evaluation and provide new empirical grounding for assessing LLM-generated tests.

This study addresses the challenges of equivalent mutant detection—namely high computational cost, compiler dependency, scarcity of labeled training data, and limited cross-language generalization—by systematically evaluating the effectiveness of large language models (LLMs) for this task across multiple programming languages. Leveraging a large-scale dataset of Java and C mutant pairs and employing a fine-tuned code embedding strategy, the proposed LLM-based approach is rigorously compared against traditional code analysis and conventional machine learning methods. Experimental results demonstrate that the LLM method significantly outperforms existing techniques in terms of accuracy (achieving the highest F1 score), inference efficiency (comparable to machine learning models), and cross-language generalization, thereby establishing its feasibility and superiority for equivalent mutant detection.