🤖 AI Summary
The absence of systematic benchmarks hampers rigorous evaluation of quantum software testing techniques. Method: We conduct a large-scale empirical study, generating over 700,000 faulty mutants from 382 real-world quantum circuits, and systematically analyze how circuit characteristics (e.g., depth, gate count), algorithm classes (e.g., QAOA), and mutation operators influence fault detection capability. Contribution/Results: We present the first quantitative characterization of the relationship between quantum circuit features and mutation detection difficulty; propose a configurable, cost-benefit–driven paradigm for constructing fault benchmarks; and release QFaultBench—an open-source tool that intelligently recommends mutants based on algorithm category and detection difficulty. Our work delivers a standardized benchmark suite, highly discriminative mutation operator combinations, and empirically grounded recommendation strategies, significantly enhancing the scientific rigor and efficiency of quantum testing benchmark construction.
📝 Abstract
As a new research area, quantum software testing lacks systematic testing benchmarks to assess testing techniques' effectiveness. Recently, some open-source benchmarks and mutation analysis tools have emerged. However, there is insufficient evidence on how various quantum circuit characteristics (e.g., circuit depth, number of quantum gates), algorithms (e.g., Quantum Approximate Optimization Algorithm), and mutation characteristics (e.g., mutation operators) affect the detection of mutants in quantum circuits. Studying such relations is important to systematically design faulty benchmarks with varied attributes (e.g., the difficulty in detecting a seeded fault) to facilitate assessing the cost-effectiveness of quantum software testing techniques efficiently. To this end, we present a large-scale empirical evaluation with more than 700K faulty benchmarks (quantum circuits) generated by mutating 382 real-world quantum circuits. Based on the results, we provide valuable insights for researchers to define systematic quantum mutation analysis techniques. We also provide a tool to recommend mutants to users based on chosen characteristics (e.g., a quantum algorithm type) and the required difficulty of detecting mutants. Finally, we also provide faulty benchmarks that can already be used to assess the cost-effectiveness of quantum software testing techniques.
