Using Fourier Analysis and Mutant Clustering to Accelerate DNN Mutation Testing

2025-10-03

AI Summary

To address the high computational cost of mutation testing for deep neural networks (DNNs)—stemming from evaluating vast numbers of mutants on large-scale datasets—this paper proposes a Fourier analysis–based acceleration method. We introduce the first application of the Fourier transform to model the spectral characteristics of mutant output behavior; spectral-space clustering is then employed to select representative mutants, enabling accurate estimation of the full-mutant score without exhaustive evaluation. Our approach achieves an average speedup of 28.38% across 14 mainstream DNN models, with only 0.72% error in mutant score estimation. Compared to random sampling and boundary-sample selection, it reduces estimation error by up to 114×. The core contribution lies in modeling mutant behavior in the frequency domain, thereby enabling high-fidelity, low-overhead approximation of full-mutant assessment.

Technology Category

• Machine Learning: Evaluation and Analysis
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Other Foundations of Computer Vision

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Deep neural network (DNN) mutation analysis is a promising approach to evaluating test set adequacy. Due to the large number of generated mutants that must be tested on large datasets, mutation analysis is costly. In this paper, we present a technique, named DM#, for accelerating DNN mutation testing using Fourier analysis. The key insight is that DNN outputs are real-valued functions suitable for Fourier analysis that can be leveraged to quantify mutant behavior using only a few data points. DM# uses the quantified mutant behavior to cluster the mutants so that the ones with similar behavior fall into the same group. A representative from each group is then selected for testing, and the result of the test, e.g., whether the mutant is killed or survived, is reused for all other mutants represented by the selected mutant, obviating the need for testing other mutants. 14 DNN models of sizes ranging from thousands to millions of parameters, trained on different datasets, are used to evaluate DM# and compare it to several baseline techniques. Our results provide empirical evidence on the effectiveness of DM# in accelerating mutation testing by 28.38%, on average, at the average cost of only 0.72% error in mutation score. Moreover, on average, DM# incurs 11.78, 15.16, and 114.36 times less mutation score error compared to random mutant selection, boundary sample selection, and random sample selection techniques, respectively, while generally offering comparable speed-up.

Problem

• Accelerating DNN mutation testing using Fourier analysis
• Clustering mutants by behavior to reduce testing costs
• Reducing mutation score error while improving speed-up

Innovation

• Uses Fourier analysis for mutant behavior quantification
• Clusters mutants based on quantified behavior similarity
• Selects representatives for testing to reduce workload